Basic MSI: Setup hangs, Leaked MSIHANDLE (38) & IScript CA

[Holger_G]

Hi,

I am using a InstallScript CA in my Basic MSI setup during UI sequence to call a standard dll.

Problem:
Installation hangs sporadically after running that CA. Sometimes with a Data Execution Prevention error of Windows Installer.

The log file reports this:[SIZE="2"]
InstallShield 13:51:43: CallScriptFunctionFromMsiCA() ends
InstallShield 13:51:43: Failed to invoke __ISWIUnInit, error is 0x8000ffff
InstallShield 13:51:43: Failed to invoke __ISRTUninit, error is 0x8000ffff
InstallShield 13:51:43: Failed to shutdown script engine for script C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\setup.inx, error is 0x8000ffff
MSI (c) (F4!C8) [13:51:43:278]: Leaked MSIHANDLE (38) of type 790541 for thread 5064
InstallShield 13:51:43: Cleaning up temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\ISRT.dll
InstallShield 13:51:43: Could not delete temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\ISRT.dll
InstallShield 13:51:43: Cleaning up temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\IsConfig.ini
InstallShield 13:51:43: Cleaning up temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\String1031.txt
InstallShield 13:51:43: Cleaning up temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\_isres.dll
InstallShield 13:51:43: Could not delete temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\_isres.dll
InstallShield 13:51:43: Cleaning up temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\_isres.dll
InstallShield 13:51:43: Could not delete temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\_isres.dll
InstallShield 13:51:43: Cleaning up temp file C:\DOCUME~1\test\LOCALS~1\Temp\{1332EB2E-46CD-4880-8EFB-A5B9D66028E1}\setup.inx[/SIZE]

I presume that the problem is caused by calling that standard .dll from inside the InstallScript CA.

Maybe you can help?

Thanks
Holger

[RobertDickau]

Could you post a sketch of what the code looks like?

[Holger_G]

Sure...here we go

[CODE]
prototype cdecl LONG TEST.TestCommandA (LPSTR , LPSTR , LPSTR );


STRING szParameters[2048], szCommand[256];
STRING szResult(256), szValue;


nvSize = MAX_PATH + 1;
if (ERROR_SUCCESS = MsiGetProperty(hMSI, "SUPPORTDIR", szSUPPORTDIR, nvSize)) then
nvSize = MAX_PATH + 1;
szDLL = szSUPPORTDIR ^ "Test.dll";
LongPathToShortPath(szDLL);
if (0 = UseDLL(szDLL)) then
szCommand = "GetType";
szParameters = "Type=BA";
szResult = "";
nReturn = TestCommandA(&szCommand, &szParameters, &szResult);
if (0==nReturn) then
Sprintf(szValue, "%s", &szResult);
nResult = MsiSetProperty(hMSI, "PTY_TYPE" , szValue);
endif;
nResult = UnUseDLL("Test.dll");
else
SprintfMsiLog("Custom Action TEST: Cannot load %s", szDLL);
endif;
else
SprintfMsiLog("Custom Action TEST: Could not get SUPPORTDIR property");
endif;
[/CODE]

[RobertDickau]

Are you returning ERROR_SUCCESS at the end of the InstallScript custom action?

(Probably doesn't matter, but does it make a difference if you use BYVAL STRING and BYREF STRING in your DLL prototype---based on the parameter names, use BYVAL for the first two and BYREF for the third?---and then pass strings instead of string pointers as arguments?)

[joshstechnij]

There could be a few causes for the hang. However, based on the script code, the DLL may be failing to unload with UnUseDLL for some reason, possibly due to an issue with DllMain in the DLL. If there's any COM loaded by the DLL, that could also pose an issue.

Regarding the DEP issues, any use of older (pre-VS 2005) ATL libraries, and anything that attempts to execute code on the stack or the heap will encounter DEP crashes on MSI 4.0 (Vista/Server 2008) and MSI 4.5 since these versions of Windows Installer are built with an /nxcompat switch that indicates to Windows that no code run by the MSI binaries will attempt to execute code in a data area of memory. A buffer overflow could also produce a DEP crash or a hang.

It may be possible that a DEP crash is causing the hang also, depending on what is occurring in the DLL.

Note the exception code from the log indicates a catastrophic failure. This means that the script engine likely started running the script code, encountered an exception, and therefore shut down.

Based on this information, you may try debugging the DLL to ensure it is working as expected, and also try Robert's suggestion to rule out parameter types as a possible cause.

[Holger_G]

Thank you both. That helps a lot.

I´ve changed the DLL prototype and return ERROR_SUCCESS.
Now it´s working without any errors on MSI 3.1. After installing MSI 4.5 to the same machine the log reports:
InstallShield 12:02:23: CallScriptFunctionFromMsiCA() ends
MSI (c) (A0:30) [12:02:23:788]: NOTE: custom action PCLE_PIXIE_GETUSERVALUES unexpectedly closed the hInstall handle (type MSIHANDLE) provided to it. The custom action should be fixed to not close that handle.

Could that be related to the /nxcompat switch build of MSI 4.5?

[joshstechnij]

The closed handle is caused by the InstallScript engine. However, it will not adversely affect the installation process and can be ignored.

[Holger_G]

joshstechnij wrote:
The closed handle is caused by the InstallScript engine. However, it will not adversely affect the installation process and can be ignored.

Will that be fixed somehow in the future?

[joshstechnij]

The closed handle issue with InstallScript custom actions logged on MSI 4.x and newer should be resolved in the next release of InstallShield.