A few pieces of information that will be helpful:
1. For what you're looking to do, you'll probably want to use the AdminStudio editor to create the transform--it exposes more granular MSI functionality.
2. For removing SelfReg entries, you can do this by unchecking the 'Self Register' option in the file's properties. To still get the file registered, use the 'COM Registration' view to extract the COM data. This will be a sub-view of 'Setup Design' or 'Components'.
3. When you add new files in a transform, it should ask you if you'd like to stream it into a cab inside the *.mst.
4. ICE errors are documented in our helpnet as well as MSDN. If you click on the ICE error when validating, it should link to some information on how to resolve it.