Old post , but same issue happens with Notes 8.5.2
I am using Adminstudio Tuner (not the one provided by IBM)
Even that in the transform was redirected to install in a different path, when running the transform program was installed in default path
issue was solved using this command:
C:\>msiexec /i "Path_to_your.msi" PROGDIR="C:\Program Files (x86)\Notes\" TRANSFORMS "Path_to_your.mst"
In other words, my MST is used for all other customization, and from the command line the property PROGDIR is redirected