Generally, languages that the installer engine doesn't natively support (and InstallShield doesn't provide a solution to support) require you to either use script or a SystemSearch/EXE custom action to find the command line interpreter (probably javaw.exe, yes?) and pass in the path to the file to execute.
This solution prevents you from accessing the MSI API, which lets you do things like read/set property values, write to the log file, etc. If you're just starting this is probably not that onerous of a limitation.
If this is a dealbreaker, there's the enticing possibility of finding (or writing) an ActiveScript bridge to Java, which would let you host the Java engine under Msiexec.exe. But that's expert level stuff, and I've never found any pre-existing ActiveScript bridge that did anything but crash when loading under Msiexec.exe. (There's a couple of exceptions: VBScript and Javascript, both of which are natively supported already...)
