I have asked this same question to the InstallShield tier 2 tech support. There is a service pack available now for 11.5 but 12.0 is not in the near future. Turns out the same development team works on both code bases for InstallShield and InstallA...
We have a dynamic suite that contains about 50 assemblies and haven't run into out of memory errors. Have you increased the max heap size the installer JVM uses? By default it is set to the Sun default of 64MB.