Im using a basic MSI project and Ive written a custom error dialog in VB which is displayed when an error occurs during an installation. The problem is the dialog is shown behind the Installshield dialog when an error occurs in the execute sequence. For errors occuring in the UI sequence the custom error dialog is shown on top of the Installshield dialog (this is correct).
At the moment Ive set the startup position in the VB project as Windows default, which should display the error dialog in the top left hand corner.
We had a similar problem with InstallScript dialogs/message boxes with Basic MSI setups. The issue is caused by incorrect parenting of the dialog/message box being displayed. The Windows Installer dialogs in a Basic MSI installation are always displayed with a higher z-order priority that almost all other windows. As a result, if you do not use the MSI dialogs as your dialog's parent/owner, there is a good chance your dialog won't always be displayed on top of the installer dialogs.
We were able to resolve the issue with InstallScript dialogs and message boxes by searching for an open MSI dialog through the FindWindow API, and, for example, pass the returned window handle in the MessageBox API's hWnd parameter.
Outside of the above option and the one mentioned by Darain, you could try using the SetWindowPos API (or its equivalent in VB) to adjust z-ordering.
