Calling a DLL from Installation via Custom Action

[Superfreak3]

Hello Again,

I'm trying to test calling a dll that verifies some user entered codes saved in Windows Installer Properties.

I've tried various options in creating the custom action, but nothing is working - can't find entry, 1723 errors, etc. I didn't log the install, but can do so.

The dll is a C++ library, I believe and basically does the following in the CompanyCheckEntries(MSIHANDLE msi)...

Checks the length of an entry stored in and grabbed from a PROPERTY and compares two other entries, also stored and grabbed from properties.

Here is some of the code...

[CODE]DLLEXPORT UINT _stdcall CompanyCheckEntries(MSIHANDLE msi)
{
CString csSerial, csAuth, csSeats;
int iRet = 0;
int iNumSeats = 0;
char buff[256];
DWORD dwSize = sizeof(buff);

int iWinErr = MsiGetProperty(msi, APP_SERIAL_NUMBER, buff, &dwSize);
if(iWinErr == ERROR_SUCCESS
&& dwSize == APP_SERIALNUMLEN ) //make sure its 8 characters
{
buff[APP_SERIALNUMLEN] = 0;
NfmTrim(buff); //comes in space padded - make sure its 8 characters
if(strlen(buff) == APP_SERIALNUMLEN)
{
csSerial = buff;
dwSize = sizeof(buff);
iWinErr = MsiGetProperty(msi, APP_AUTHORIZATION_CODE, buff, &dwSize);
}
}
if(iWinErr == ERROR_SUCCESS)
{
csAuth = buff;
dwSize = sizeof(buff);
iWinErr = MsiGetProperty(msi, APP_NUMBER_OF_SEATS, buff, &dwSize);
}
if(iWinErr == ERROR_SUCCESS)
{
csSeats = buff;
dwSize = sizeof(buff);
iNumSeats = atoi(buff);
}

int iSetValid = -1;
if(iNumSeats > 0)
{
iRet = ERROR_SUCCESS;
Company_SIGNATURE adSig, encSig;
memset(&adSig,0,sizeof(adSig));
memset(&encSig,0,sizeof(encSig));

strcpy(adSig.signature,STD_SIGNATURE);
NfmSafeCopy(adSig.serial, csSerial, AD_SERLEN );
NfmSafeCopy(adSig.authcode, csAuth, AD_AUTHLEN );
NfmSafeCopy(adSig.machine, "0", AD_MACHLEN);
NfmSafeCopy(adSig.socket, "0", AD_SOCKLEN);
int iSeats = EncryptSignature( &adSig, &encSig );
MsiSetProperty(msi,APP_VALID, iSeats == iNumSeats ? "1" : "0");
}

return iRet;
}[/CODE]

With all of this, what action should I use - Standard .dll or Windows Installer .dll? Walking through the CA Wizard, how would I set this up?

Basically what happens then is a dialog is shown indicating invalid codes if APP_VALID = "0" when the Dialog's Next button is pressed. Otherwise (="1") the interface continues.

Any help with this would be greatly appreciated!

[RobertDickau]

If the DLL functions have the signature

UINT __stdcall SomethingSomething(HWND hInstall)

you'll want to use a Windows Installer (MSI) DLL action and not a "standard DLL" action.

Since these are called in the user interface, you can define the action to use immediate mode. Since you're triggering the action with a button click, the dialog should use a DoAction control event to trigger the action, instead of scheduling the action in the sequences.

If you use Dependency Walker or dumpbin, are the exported function names what you expect? If not, you might want to build the DLL using a .def file, which suppresses C++ name decoration for the exported functions.

(It's not central to the issue, but it appears the function will never return anything but the value ERROR_SUCCESS, which is 0, even though there's logic in there to set the return value.)

[Superfreak3]

The code does work in our current installations built with the product we are looking to migrate from.

I'll keep hacking away at it.

I guess I'll have to look into the naming issue a bit.

I know how to set up the action from the dialog, but I was wondering what setting the actual Custom Action needed with regard to the dll itself (function called, which I will look into the naming, etc).

Oh, thanks for the Dependency Walker idea - it gave me _CompanyCheckEntries@4 as the Function Name.