CustomAction fails on Vista only, except when run in IDE

[sjdworkin]

I upgraded an older IS 10.5 Basic MSI project to IS2008 so that we could run the install on Vista machines. Setup.exe still works fine on XP, but on Vista it fails. Windows Installer returns a 1603 (non-specific error) and logs the message "Custom action MoveConfigFile unexpectedly closed the hInstall handle (type MSIHANDLE) provided to it. The custom action should be fixed to not close that handle."

I've seen that message in enough posts on this forum to realize that it doesn't mean that the InstallScript in our custom action explicitly closed a handle. (See code below.) It means that some unknown exception caused the IScript engine to abort the call.

So I wanted to see how it behaved in the debugger. I tried to run the (properly registered) remote debugger ISDbg.exe on the Vista machine, but the debugger didn't come up during the run. (It came up fine on the XP machine.) Then I temporarily installed the IS IDE on the Vista machine and ran the project in Debug mode. Again it didn't bring up the debugger when it ran the custom actions (though it does so on XP). But the funny thing was that, run this way on Vista, the install worked!

My only guess is that the IDE runs the action with higher permissions than when I run the same Setup.exe standalone. Any suggestions as to what I could fix in the custom action to get it to run on Vista outside of the IDE?

Code:

function MoveConfigFile(hMSI)
NUMBER nResult;
STRING svSetupDir,svPersonalDir;
STRING origFile, dstFile, svResult;
INT nSize;
NUMBER nvResult;
begin
nSize = 256;

MsiGetProperty(hMSI,"SETUPEXEDIR", svSetupDir, nSize);
origFile = svSetupDir ^ "\\axssetup\\axsdesktop.xml";
//copy the file to the install dir and the user profile directory
dstFile = INSTALLDIR ^ "axsdesktop.xml";
if (GetFileInfo ( origFile, FILE_SIZE, nvResult, svResult )= 0) then

// Copy all files in the source directory, including files
// in subdirectories, to the target directory.
nResult = CopyFile(origFile, dstFile);

// Report the results of the copy operation in installdir.
switch (nResult)
case 0:
//MessageBox ("Files successfully copied.", INFORMATION);
return 0;
case COPY_ERR_CREATEDIR:
MessageBox ("A target directory could not be created.", SEVERE);
case COPY_ERR_MEMORY:
MessageBox ("Insufficient memory.", SEVERE);
case COPY_ERR_NODISKSPACE:
MessageBox ("Insufficint disk space.", SEVERE);
case COPY_ERR_OPENINPUT:
MessageBox ("Unable to open the input files in "+ origFile +".",
SEVERE);
case COPY_ERR_OPENOUTPUT:
MessageBox ("Unable to copy the source files.", SEVERE);
case COPY_ERR_TARGETREADONLY:
MessageBox ("A target file already exists and cannot be overwritten.",
SEVERE);
default:
MessageBox ("An unspecified error occurred.", SEVERE);
endswitch;

else
SprintfBox (INFORMATION, "Installation failed to find file",
"Configuration file:\n\n%s\n\n",origFile );
return -1;
endif;

return nResult;
end;

Thanks -- Steve

[MichaelU]

Generally a target directory of INSTALLDIR will require administrative privileges to write to (unless you have written a per-user installation), so this action should be scheduled as Deferred in System Context. Making that work correctly will require some changes as described in the migration documents for the IS12 engine refactor (see this and the IS12 community, or the KB). A stopgap measure which is usually not the best solution would be to set a manifest on setup.exe to require administrator privileges to run the setup; see the appropriate tab on the release grid for your release. This would replicate the elevated installation conditions you see when launching from the elevated IDE.

[sjdworkin]

That makes sense. I'll make it deferred, and I did already see the doc on the differences for deferred actions in IS12.

[sjdworkin]

Actually, all I had to do was right-click on Setup.exe and select Run As Administrator, everything worked. Thanks for steering me in the right direction... Steve