InstallScript Custom Action issue

[IlkkaV]

Hi

We're in the process of migrating our InstallShield 2009 project (originally migrated from IS 12 through IS 2008) to InstallShield 2010, but there's currently one showstopper regarding custom actions. The project itself is basic MSI, but custom actions are written using InstallScript. Now some of the custom actions fail when trying to extract support file String0000.txt (the purpose of which is unknown to me). Installation log has the following:

MSI (s) (60:C0) [11:39:53:372]: Invoking remote custom action. DLL: C:\Windows\Installer\MSID044.tmp, Entrypoint: f10
InstallShield 11:39:54: Running InstallScript function f10
InstallShield 11:39:54: Opening stream of file C:\Windows\Installer\MSID044.tmp
InstallShield 11:39:54: Extracting support file IsConfig.ini to C:\Users\admin\AppData\Local\Temp\{07D62BE0-86C4-437C-9023-EE7A957D6CA0}\IsConfig.ini
InstallShield 11:39:54: Extracted isconfig.ini to C:\Users\ilkviin\AppData\Local\Temp\{07D62BE0-86C4-437C-9023-EE7A957D6CA0}\IsConfig.ini
InstallShield 11:39:54: Got '{28FED039-E987-496D-900D-D8557D44D589}' for TempPathGuid from isconfig.ini
InstallShield 11:39:54: Attempting to use temp path 'C:\Users\admin\AppData\Local\Temp\{28FED039-E987-496D-900D-D8557D44D589}'
InstallShield 11:39:54: Using new temp path
InstallShield 11:39:54: Cleaning up temp file C:\Users\admin\AppData\Local\Temp\{07D62BE0-86C4-437C-9023-EE7A957D6CA0}\IsConfig.ini
InstallShield 11:39:54: Using temp folder C:\Users\admin\AppData\Local\Temp\{28FED039-E987-496D-900D-D8557D44D589}
InstallShield 11:39:54: Installing engine...
InstallShield 11:39:54: Using product language 0
InstallShield 11:39:54: Skipping optional support file _isuser_0x0409.dll
InstallShield 11:39:54: Setting script cmdline...
InstallShield 11:39:54: ProductCode is {E408CD93-A5A9-45FD-9B3F-581A846161BA}
InstallShield 11:39:54: Initializing Engine
InstallShield 11:39:54: Extracting support file String0000.txt to C:\Users\admin\AppData\Local\Temp\{28FED039-E987-496D-900D-D8557D44D589}\String0000.txt
InstallShield 11:39:54: Failed to extract file String0000.txt. Error = 0
InstallShield 11:39:54: Done Initializing...
InstallShield 11:39:54: Registering Msi Server...

I noticed that all but one of the functions that cause a similar error utilize CustomActionData (and basically every CustomActionData function that gets called seems to fail) and the one remaining tried to read a value from the Registry (also runs LaunchAppAndWait). On XP the error is logged but the functions still seem to work. However, on Vista and 2008 Server the issue is a few magnitudes worse, as multiple "Windows Installer has stopped working" errors are spawned and the functions don't do anything. Any ideas on how to resolve this?

[joshstechnij]

The failure to extract string0000.txt is a non-fatal error as evidenced by the proceeding log lines "Done initializing" and "Registering MSI server". There should be additional log information after these lines indicating what the InstallScript engine is attempting to do. Can you post the information in the log up to the next action start message? Also, can you reproduce this behavior in a sample project that uses similar script code in an InstallScript custom action (I haven't been able to reproduce any failures with simple InstallScript custom actions)?

[IlkkaV]

I noticed that on Vista/2008 there is an additional "Lost connection to custom action server process. Attempting to regenerate" message. In addition, on XP a custom log message written by SprintfMsiLog is included in the log, but that's not the case with Vista/2008, so it seems like the function is not run at all on Vista/2008.

Vista log:

MSI (s) (DC:A0) [08:09:43:033]: Executing op: ActionStart(Name=RunTransferTool,,)
MSI (s) (DC:A0) [08:09:43:083]: Executing op: CustomActionSchedule(Action=RunTransferTool,ActionType=3137,Source=BinaryData,Target=f8,)
MSI (s) (DC:64) [08:09:43:264]: Invoking remote custom action. DLL: C:\Windows\Installer\MSI49E5.tmp, Entrypoint: f8
MSI (s) (DC:64) [08:09:43:264]: Lost connection to custom action server process. Attempting to regenerate.
MSI (s) (DC:08) [08:09:43:264]: Generating random cookie.
MSI (s) (DC:08) [08:09:43:354]: Created Custom Action Server with PID 2640 (0xA50).
MSI (s) (DC:9C) [08:09:43:774]: Running as a service.
MSI (s) (DC:9C) [08:09:43:784]: Hello, I'm your 32bit Elevated custom action server.
InstallShield 8:09:44: Running InstallScript function f8
InstallShield 8:09:44: Opening stream of file C:\Windows\Installer\MSI49E5.tmp
InstallShield 8:09:44: Extracting support file IsConfig.ini to C:\Users\TESTMA~1\AppData\Local\Temp\{01A60FF4-5C31-4E55-B824-553FB53FEA86}\IsConfig.ini
InstallShield 8:09:44: Extracted isconfig.ini to C:\Users\TESTMA~1\AppData\Local\Temp\{01A60FF4-5C31-4E55-B824-553FB53FEA86}\IsConfig.ini
InstallShield 8:09:44: Got '{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}' for TempPathGuid from isconfig.ini
InstallShield 8:09:44: Attempting to use temp path 'C:\Users\TESTMA~1\AppData\Local\Temp\{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}'
InstallShield 8:09:44: Using new temp path
InstallShield 8:09:44: Cleaning up temp file C:\Users\TESTMA~1\AppData\Local\Temp\{01A60FF4-5C31-4E55-B824-553FB53FEA86}\IsConfig.ini
InstallShield 8:09:44: Using temp folder C:\Users\TESTMA~1\AppData\Local\Temp\{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}
InstallShield 8:09:44: Installing engine...
InstallShield 8:09:44: Using product language 0
InstallShield 8:09:44: Skipping optional support file _isuser_0x0409.dll
InstallShield 8:09:44: Setting script cmdline...
InstallShield 8:09:44: ProductCode is {E408CD93-A5A9-45FD-9B3F-581A846161BA}
InstallShield 8:09:44: Initializing Engine
InstallShield 8:09:44: Extracting support file String0000.txt to C:\Users\TESTMA~1\AppData\Local\Temp\{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}\String0000.txt
InstallShield 8:09:44: Failed to extract file String0000.txt. Error = 0
InstallShield 8:09:44: Done Initializing...
InstallShield 8:09:44: Registering Msi Server...
MSI (s) (DC:A0) [08:09:51:095]: Executing op: ActionStart(Name=RegisterUser,Description=Registering user,Template=[1])


XP log:

MSI (s) (50:34) [08:45:15:102]: Executing op: ActionStart(Name=RunTransferTool,,)
MSI (s) (50:34) [08:45:15:112]: Executing op: CustomActionSchedule(Action=RunTransferTool,ActionType=3137,Source=BinaryData,Target=f8,)
MSI (s) (50:6C) [08:45:15:142]: Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSI9B.tmp, Entrypoint: f8
InstallShield 8:45:15: Running InstallScript function f8
InstallShield 8:45:15: Opening stream of file C:\WINDOWS\Installer\MSI9B.tmp
InstallShield 8:45:15: Extracting support file IsConfig.ini to C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{B88CEBCD-2582-46AC-9EA7-97AD7437D785}\IsConfig.ini
InstallShield 8:45:15: Extracted isconfig.ini to C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{B88CEBCD-2582-46AC-9EA7-97AD7437D785}\IsConfig.ini
InstallShield 8:45:15: Got '{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}' for TempPathGuid from isconfig.ini
InstallShield 8:45:15: Attempting to use temp path 'C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}'
InstallShield 8:45:15: Using new temp path
InstallShield 8:45:15: Cleaning up temp file C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{B88CEBCD-2582-46AC-9EA7-97AD7437D785}\IsConfig.ini
InstallShield 8:45:15: Using temp folder C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}
InstallShield 8:45:15: Installing engine...
InstallShield 8:45:15: Using product language 0
InstallShield 8:45:15: Skipping optional support file _isuser_0x0409.dll
InstallShield 8:45:15: Setting script cmdline...
InstallShield 8:45:15: ProductCode is {E408CD93-A5A9-45FD-9B3F-581A846161BA}
InstallShield 8:45:15: Initializing Engine
InstallShield 8:45:16: Extracting support file String0000.txt to C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{A5358B20-E2F4-4F35-9C03-6F12F5AB58F7}\String0000.txt
InstallShield 8:45:16: Failed to extract file String0000.txt. Error = 0
InstallShield 8:45:16: Done Initializing...
InstallShield 8:45:16: Registering Msi Server...
InstallShield 8:45:16: Invoking script function RunTransferTool
1: Calling RunTransferTool
InstallShield 8:45:53: CallScriptFunctionFromMsiCA() ends
MSI (s) (50:34) [08:45:53:818]: Executing op: ActionStart(Name=RegisterUser,Description=Registering user,Template=[1])

I haven't been able to reproduce this on a sample project yet, but I'll keep trying.

[joshstechnij]

The line indicating "lost connection to custom action server process" is logged by Windows Installer before any of our code is invoked. Unfortunately, we do not have any information on what could cause this behavior. However, in the interests of verifying nothing on the machine is interfering with the installation process, please ensure all anti-virus, anti-malware, anti-spyware or similar utilities are disabled before testing the installation. We have seen issues in the past with InstallScript custom actions caused by these types of utilities flagging the engine files as malicious.

[IlkkaV]

joshstechnij wrote:
However, in the interests of verifying nothing on the machine is interfering with the installation process, please ensure all anti-virus, anti-malware, anti-spyware or similar utilities are disabled before testing the installation. We have seen issues in the past with InstallScript custom actions caused by these types of utilities flagging the engine files as malicious.

All those utilities were off on both machines (both virtual). In addition, earlier versions of the project (still built with IS 2009) install on both without any issues. Could it be some glitch in project conversion?

[joshstechnij]

There haven't really been any major changes to InstallScript custom actions in IS 2010 that could cause this (project migration makes minor changes related to InstallScript library information in the ISLinkerLibrary table to remove path references).

Can you try launching your setup from the MSI package instead of setup.exe? Does this change the behavior you are seeing?

[IlkkaV]

joshstechnij wrote:
Can you try launching your setup from the MSI package instead of setup.exe? Does this change the behavior you are seeing?

Tried that, but unfortunately it didn't help. However, as we have multiple languages and setup prerequisites included in the installer, we'd need to have a setup.exe anyway.

[joshstechnij]

I am able to reproduce this issue now with the IS 2010 release. This appears to only reproduce for me on 64-bit machines with deferred in system context InstallScript custom actions. Were you testing this project on 64-bit machines?

[joshstechnij]

After researching this further, it appears that the following conditions are needed to encounter this issue:
- Multi-language project
- InstallScript custom actions set to run deferred in system context
- Installation is run on a 64-bit machine (possibly Vista or 2008 specific)

We should have a fix posted for this early next week (I've submitted work order IOC-000081969 to track this issue).

[IlkkaV]

Both of my test machines were 32-bit editions run on Virtual PC 2007. The host machine (this workstation), is a 64-bit 2008 Server, and sure enough the problem could be reproduced on that as well. Nevertheless, it's great to hear that a fix is coming 🙂

[IlkkaV]

Will there be an announcement here when the fix is available or should I monitor some other page?

[joshstechnij]

I'll post the fix in this thread and a KB article will eventually be made available.

[joshstechnij]

I have attached a hotfixed version of ISSetup.dll. To apply this fix:
1. Make a backup copy of the file C:\Program Files\InstallShield\2010\Redist\Language Independent\i386\ISSetup.dll.
2. Extract the attached zip file to C:\Program Files\InstallShield\2010\Redist\Language Independent\i386.
3. Rebuild any project that uses InstallScript custom actions.

Please let us know if you encounter any issues with this fix.

[IlkkaV]

Thanks Josh! Worked like a charm on Vista (have to still check on XP, though). I guess the same hotfix is good to go with SAB, too?

[joshstechnij]

If you have the standalone build, the same procedure to apply the fix to the IDE will work (the paths and everything are the same in the SAB since IS 2009).

[IlkkaV]

Excellent. XP worked fine, too, so I'll start setting up my 2010 SAB environment. Thank you!

[Alibaba]

I get the same error (Lost connection to custom action server process...) in an installshield 2009 project but with an immediate execution CA. Can I also apply the patch on a IS2009?