Setup EXE Fails to Launch - Long Pathname Issue
Recently we faced an issue where one of our software product installers was unable to run and install the application on certain Windows 10 computers. To deploy our application, we use a basic MSI project with an InstallShield compressed Setup EXE wrapper.
The Setup EXE should open in the ideal circumstance, and after allowing the UAC prompt, the prerequisite installation and MSI installation should proceed normally.
The issue can be seen once the Setup EXE is launched and the UAC prompt appears, but nothing happens after that. The MSI database is not even extracted, and no prerequisites are installed. The EXE process terminates just after few seconds and the /debuglog parameter has nothing.
After further investigation, we discovered that the problem only occurs on certain computers and with certain individuals that have a whitespace in their username. The application installs correctly if we manually copy and run the extracted MSI database on these computers. The Setup EXE, on the other hand, fails to install on the same system.
We manually changed the %TEMP% and %TMP% environment variables to a valid path with no spaces and discovered that the Setup EXE could execute the installer. Additionally, enclosing the existing %TEMP% or %TMP% in quotes ("") resolves the issue.
We spoke with Flexera Support about this issue, and after more investigation, we discovered that on the faulty workstations, we are also unable to open %TEMP% or %USERPROFILE% using the Run command. This led to the conclusion that this is a machine-specific issue that has nothing to do with whitespace or user accounts.
After further investigation, we found the root cause, the NtfsDisable8dot3NameCreation key present in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem. This key is responsible for converting windows long path names to appropriate short path name notation so that the applications can function properly. This is set to 2 (Enabled) by default on all Windows 10 computers, however it can be discovered set to 1 (Disabled) on some systems that have the problem. As a result, the application is unable to properly read the %USERPROFILE% and %TEMP% locations. After we've properly set this key, we'll need to create a new user on the machine and then use that user to run the Setup EXE to resolve the issue. It won't work for existing users of the machine.
Our question is:
Is there any way to detect this registry key and check if the value is incorrect through the Setup EXE launcher? If it is incorrect, then show a proper message box to our end users?
We know we can do that in our MSI database, but the MSI database isn't even getting called in this scenario because the Setup EXE fails to launch. We'd like to handle this scenario within the Setup EXE launcher's process.
Any help or suggestion would be greatly appreciated.