cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
christian_wach
Level 2

Registering a dll fails

Jump to solution

We have a dll for an add on which is installed (into a sub folder of the main tool) and must be registered (from an IS script calling regsvr32). To be able to load our dll it needs another dll from the main tool so our IS script cd's into the main tool installation folder and then calls "regsvr32 /s <full path of our dll>". regsvr32 can load our dll because the current dir is the main tool dir where the needed dll can be found. We provide a merge module for our tool which is integrated into the installer (MSI) of the main tool.

This was working for a long time but suddenly it doesn't work anymore: Now it still works on Windows 7 while on Windows 8 and 10 it only works if we add the main tool's installation path to the environment PATH. Otherwise regsvr32 fails to load our dll because it cannot find the dll from the main tool (but the dll is definitely there).

The developer of the main tool switched to IS 2018 R2 recently - it might be related to the issue.

Our IS script is basically doing this:

 

ChangeDirectory(szMainToolDir);

szDllPath = <full path to our dll>;

LongPathToQuote(szDllPath, TRUE);

szDllPath = "/s " + szDllPath;

LaunchAppAndWait("regsvr32", szDllPath, LAAW_OPTION_WAIT | LAAW_OPTION_HIDDEN);

-> This fails, regsvr32 returns error code 3 (the dll could not be loaded).

Labels (1)
0 Kudos
(1) Solution
Varaprasad
Level 7 Flexeran
Level 7 Flexeran

Please check below link:

https://helpnet.flexerasoftware.com/installshield24helplib/rn/ReleaseNotes.htm#isreleasenotes_2630790174_1049156

Current Directory Removed from DLL Search Path to Safeguard Against DLL Preloading Attacks

To safeguard installations against DLL preloading attacks, InstallShield has removed the current directory from the standard DLL search path by calling the SetDllDirectory Windows API with an empty string ("").

If a DLL links implicitly to the other DLLs, or loads them dynamically using LoadLibrary() without specifying a fully qualified path name, the UseDLL() InstallScript function cannot load the dependencies from the current working directory.

You can work around this issue by prototyping SetDllDirectoryW (prototype number kernel32.SetDllDirectoryW(wstring); ) and call it with SUPPORTDIR to get the support folder to be in the DLL load search path.

In InstallShield 2018, the following changes have been made:

  • The DLL_DIRECTORY_SUPPORTDIR constant was added to the Enable function so that customers can explicitly opt-in to using SUPPORTDIR as a DLL directory.
  • The DLL_DIRECTORY_SUPPORTDIR constant was added to the Disable function so that customers can explicitly opt-out to using SUPPORTDIR as a DLL directory.
  • The SetDllDirectory (szPathName) wrapper function was added so that customers can explicitly opt-in to using any directory as a DLL directory. If the parameter is an empty string (""), the call removes the current directory from the default DLL search order.

 

View solution in original post

(3) Replies
Varaprasad
Level 7 Flexeran
Level 7 Flexeran

Please check below link:

https://helpnet.flexerasoftware.com/installshield24helplib/rn/ReleaseNotes.htm#isreleasenotes_2630790174_1049156

Current Directory Removed from DLL Search Path to Safeguard Against DLL Preloading Attacks

To safeguard installations against DLL preloading attacks, InstallShield has removed the current directory from the standard DLL search path by calling the SetDllDirectory Windows API with an empty string ("").

If a DLL links implicitly to the other DLLs, or loads them dynamically using LoadLibrary() without specifying a fully qualified path name, the UseDLL() InstallScript function cannot load the dependencies from the current working directory.

You can work around this issue by prototyping SetDllDirectoryW (prototype number kernel32.SetDllDirectoryW(wstring); ) and call it with SUPPORTDIR to get the support folder to be in the DLL load search path.

In InstallShield 2018, the following changes have been made:

  • The DLL_DIRECTORY_SUPPORTDIR constant was added to the Enable function so that customers can explicitly opt-in to using SUPPORTDIR as a DLL directory.
  • The DLL_DIRECTORY_SUPPORTDIR constant was added to the Disable function so that customers can explicitly opt-out to using SUPPORTDIR as a DLL directory.
  • The SetDllDirectory (szPathName) wrapper function was added so that customers can explicitly opt-in to using any directory as a DLL directory. If the parameter is an empty string (""), the call removes the current directory from the default DLL search order.

 

Thanks a lot, this helps.

If we want to enable DLL_DIRECTORY_SUPPORTDIR: Can this be done in our merge module or must it be done in the main tool's MSI project?

0 Kudos

Calling SetDllDirectory(szMainToolDir) before LaunchAppAndWait method should work for you. I would recommend to revert to the standard search path by calling SetDllDirectory("") to safeguard your installation.

0 Kudos