I have a (test) custom action DLL written in VC++8 for my setup project. All the custom action does is "MessageBox(...); return(ERROR_SUCCESS);". I've tested it with VS2005's setup project and it works OK. However when I include it as a custom action in my IS project I get some weird behaviour.
The custom action is executed, but the installer fails saying that it has been interrupted. The log from the MSI says that it returned value 3.
Has anyone seen this problem and solved it before? Ignoring the exit code is not going to be an option for me, neither is switching back to VS2003, which are the only two suggestions I've found so far.
I thought it might have been something to do with the DLL manifests because a FileMon log shows that msiexec tried to open but failed for the embedded manifest and config files in the DLL. I turned off manifest generation but it made no difference.
I figured out my problem after re-reading the manual this morning. Despite the function prototype for the "New" mode being described as "using the standard MSI custom action signature", it doesn't. The return value is different. I wouldn't call that standard.
In a standard Windows Installer custom action DLL function the return value can be one of ERROR_FUNCTION_NOT_CALLED, ERROR_SUCCESS (which has value 0), ERROR_INSTALL_USEREXIT, ERROR_INSTALL_FAILURE or ERROR_NO_MORE_ITEMS.
IS expects that your custom action DLL function returns 0 for failure and non-zero for success.
