No, the InstallScript engine is only a 32-bit engine and so it is not able to directly load 64-bit dll's into memory.
Can I load 64-bit dll files in InstallScript?
No, the InstallScript engine is only a 32-bit engine and so it is not able to directly load 64-bit dll's into memory. So if you try and use the UseDLL() function on a 64-bit dll file, you will encounter an error at runtime.
Unfortunately, there is no convenient way around this limitation. Depending on what you need to access in your 64-bit dll, you could create a stub 64-bit executable that can load the 64-bit dll into memory and then intermediate between the 32-bit InstallScript installer context and the 64-bit DLL context (...if simple cross-communication is necessary, it can be peformed via command line arguments and/or small INI or XML etc. files).
If you create the 64-bit stub exe, you can use the LaunchApplication() function to run it from InstallScript....LaunchApplication