Still: COM Extraction of ProgIDs in IS 2009 does NOT WORK!

[Mike_7]

I am a little bit furious, because at last I found another thread, describing exactly the same problem earlier which I now face with InstallShield 2009:
http://community.installshield.com/showthread.php?t=174455
In the mentioned thread the problem arised October 2007, still there is NO SOLUTION!

No matter if we use "COM-Extraction at build" or if we extract all COM-Information of our VB6-DLLs at once within the IS-Project - there is a difference in the COM-Registration that regsvr32.exe would generate! And that causes many problems after deployment - who will pay us for that?? :mad:

To be more detailed:

The general problem is, that the ProgIDs of the COM-Server are not (completely) registered in HKEY_CLASSES_ROOT...
It is a strange behaviour: Not every COM-DLL is treated the same way:
In our project we have some 15 COM-DLLs, each created with Visual Studio 6.0.
The classes in the VB6-Project have different "Instancing"-Properties: Some are "MultiUse", some are "GlobalMultiUse", some are "PublicNotCreatable".
13 of 15 DLLs register well with COM-Extraction at build.

Situation of DLL "No. 14":
It contains three public COM-Classes.
I extracted the COM-Information to the current IS-Project. I investigated the section "Registry" below the Component-Node (I hope I translate right to the english version...). I see below HKEY_CLASSES_ROOT just the folders "Interface", "CLSID" and "TypeLib".
Nice. But the expected folders for the ProgIDs are missing.
When I open the "Advanced Settings" - "COM-Registration" for that component I find three entries below "COM-Classes" (is okay) and I find the three ProgIDs below. Okay, but the setup does DEFINITIVELY NOT REGISTER these ProgIDs! And thus the program cannot instantiate objects by using the ProgIDs.
All three classes have the "Instancing"-Property "MultiUse" in VB6.

Situation of DLL "No. 15":
It contains 18 public COM-classes that have to be registered.
6 classes with Instancing "MultiUse" (within VB-Project)
4 classes with Instancing "GlobalMultiUse"
8 classes with Instancing "PublicNotCreatable".
The funny thing is:
After extracting the COM-Information to the IS-Project I actually see the 8 classes with VB6-Instancing "PublicNotCreatable" listed below HKEY_CLASSES_ROOT in the registry-view of the component as it has to be. The other 10 classes are missing.
When I now open "Advanced Settings" - "COM-Registration" for that component: There I find the OTHER 10 classes: 10 entries below "COM-classes" and 10 entries below "ProgIDs".

I added the screenshots of this situation for the "unbelievers" as attachments... One shows the 8 ProgIDs in the registry-view that really reach the destination computer after setup, the other shows the 10 missing ProgIDs in the "Advanced Settings"-View of the component.

So now my QUESTION:
WHY do some classes register completely okay,
while other classes (that are produced the exact same way with VB6) register only the ProgIDs of the "PublicNotCreatable"-classes
:confused:

Thanks in advance.
Mike

(Please don´t tell me: "Use Self-Registration instead". It causes much more problems with uninstallations...!)

[CChong]

I know too good what you mean, Mike - but can't provide any help.

I myself have pretty always to register DirectShow filters which I still have to do and undo via regsvr32 calls. Not long ago InstallShield versions didn't even recognize .ax files as COM servers but also now that they do the built-in COM extraction doesn't suffice.
That's because the standard AMovieDllRegisterServer2() for DShow filters does much more than simple COM registration, also with the upcoming Media Foundation Transforms I don't have much hope to get rid of regsvr32. And if you look at other Windows SDKs, there are so much different registration methods around that I can understand how hard it would be to cover all of them without calling DllRegisterServer. 😞

[datamine]

This is becoming an increasing problem for us now too, and InstallShield (or whatever they choose to be called this week) don't appear to be taking it seriously. It's costing us a lot in man-hours to extract the COM entries manually for every component, with no guarantee that these entries will be future-proof. So much for "Best Practices" recommending the use of COM Extract at Build!

[joshstechnij]

Unfortunately, without any additional information, it is difficult to determine why COM data may not be picked up by COM extraction. Here is a list of possible causes:
- The DLL server in question attempts to conditionally register information based on data read from the registry. The current COM extraction implementation uses the RegOverridePredefKey API to redirect registry I/O. As a result, almost no registry data (with the exception of some ATL keys) will be present. You can try using the old COM extraction method (API hooking, which is less reliable on modern Windows platforms) by setting UseAPIRegistryHooks to 0 in HKLM\Software\InstallShield\RegySpy.
- The registry data is excluded because it is present in C:\Program Files\InstallShield\2009\Support\filters.xml as data that should not be populated in an MSI package (typically Windows protected registry keys).
- The registry data could cause issues when populated in an MSI package, and may not be included as a result (this is done to avoid situations that populate registry data such as HKLM\Software\* which will cause MSI to attempt to remove all registry data in HKLM\Software during uninstall, breaking a machine).

While there may be other causes, the above are the main reasons COM data isn't extracted. Unfortunately, there are some COM servers that just won't play nice with COM extraction for one reason or another (mostly due to conditional registration based on environment). In these cases, self registration is recommended. Alternately, if the COM data for the server in question is known, it can be populated through the MSI tables manually.

If you have any reproduction cases that do not perform things like conditional registration in their registration entry point, please let us know.

[predatorsw]

Installshield's com extraction is retarded and doesn't work. Neither pulling it using the refresh commands nor extraction at build work correctly.

If you use ATL and the ATL registrations functions (.rgs files) then it works fine. If you use OLE and AfxOleRegisterTypeLib/COleObjectFactoryEx::UpdateRegistryAll then it registers the OLE part but doesn't create the ProgID information (the dot version names).

We were using the selfreg route but this creates a mountain of problems with installs on Windows 7 and is no longer a viable option.

We're still using X so it's good to see I haven't been wasting money on newer versions that still doen't fix this blatently obvious problem.

[joshstechnij]

InstallShield is performing the same basic steps to register a COM server that regsvr32.exe would do, i.e.:
- LoadLibrary the COM server (non-EXE case)
- GetProcAddress for DllRegisterServer and call the returned function pointer

It is up to the DllRegisterServer entry point to perform the actual registration. Please see my previous post for how we attempt to capture the information written to the registry by DllRegisterServer.

[predatorsw]

ProcessMonitor (formerly RegMon) captures the registry updates just fine. Installshield doesn't. It is not dllregisterserver's problem, it's Installshields.

If regsvr32 fixes the problem then dllregisterserver works perfectly. If Installshield doesn't get out exactly the same information written to the registry when regsvr32 runs ON THE EXACT SAME MACHINE IN THE EXACT SAME FOLDER AT THE EXACT SAME TIME then Installshield is broken. Period.

I'm currently using ProcessMonitor to manually dig our com registration information out of our dll's since Installshield can't handle it. They seem to be able to capture everything just fine.

The source for RegMon is floating around out there from back in the SysInternals days. You should spend the effort looking for it instead of inventing more excuses.

I will reiterate just to make this perfectly clear. Installshield's COM extraction does not work. It is broken. It is proveably broken. It has been broken since this feature was added to Installshield. The solution is out there in source code form. All Installshield has to do is find it and use it.

[predatorsw]

You may want too google "installshield com extraction" to see just how many people have given up on installshield's ability to do this. I did find this possible work around:

http://www.installsite.org/pages/en/msi/isd.htm

which I'll give a shot because I'm bored out of my mind extracting com information manually when these tools should be able to handle it. So I can guarantee installshield up to at least 12 didn't have COM extraction that worked.

[joshstechnij]

Both Regmon and ProcessMon use a kernel registry filter driver (or system service dispatch table hooking on Windows XP) to capture registry calls on a machine (as documented in Windows Internals 4th and 5th editions) instead of user mode methods that InstallShield uses. This is something we have researched but due to limitations with the registry information provided by the configuration manager filter callback on Windows XP (which is why SSDT hooking is used by Regmon and ProcessMon on this OS), would limit the availability of this functionality to at least Windows 2003 and more likely Windows Vista or newer (though that does not rule out it being available in some future release).

Regarding DEP issues on Windows 2003 and newer machines, this was resolved with changes made to COM extraction in IS 2009 when 64-bit support was added. However, this issue is specific to COM extraction for EXE files since we need to either provide API hooks or turn on registry redirection in the new process; the memory allocated in the remote process for code injection is now allocated as executable memory allowing us to avoid any DEP issues. This issue does not occur with DLL, OCX, etc. type files that can be loaded into our process address space.

Regarding complaints of COM extraction not working on older releases of InstallShield (mostly pre-IS 12 on XP SP2 or newer machines with COM servers built in VS 2005 or newer), this is due to the API hooking that COM extraction used to rely on failing at a large rate. Since API hooking on modern versions of Windows is not as usable as it once was (possibly because of large sweeping security changes), we decided to use registry redirection with the RegOverridePredefKey API as it yielded a much higher success rate in scenarios where API hooking was no longer working. Another issue was caused in the standalone build since it used to be built on a modified codebase compared to the IDE code (this was eliminated with IS 2009 so both IDE and standalone codebases are identical) that could cause issues specific to standalone build. Registry filtering was added in IS 12 to address the new COM extraction's tendency to pick up unrelated registry keys.

Unfortunately, out of all the results provided from a Google search, no one has provided concrete examples of what is wrong when they try to use COM extraction (i.e. something that we can use to reproduce and isolate the cause of the behavior). Stating that COM extraction "isn't working", while understandable, does not provide us with sufficient information to be able to determine the cause of a failure. Our support department will occasionally receive issues with COM extraction that we resolve on a case-by-case basis, but they tend to be one-off issues. As I mentioned in an earlier post, if you have any reproduction cases, please let us know so that we can investigate further.

[CChong]

Concrete example, DirectShow filters:

If I remember it right, IShield's COM extraction gets the direct GUID-subkey and entries of HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\ and HKEY_CLASSES_ROOT\CLSID, resp., which is the "ordinary" COM part.
But the "ActiveMovie" part below HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{083863F1-70DE-11d0-BD40-00A0C911CE86}\Instance\ (resp. HKCR\CLSID\{083863F1-...}\Instance\) it doesn't extract correctly or not at all. There are more places to fetch, below {33D9A760-90C8-11d0-BD43-00A0C911CE86}\Instance (ICM Class Manager), {7D22E920-5CA9-4787-8C2B-A6779BD11781}\Instance, may be more...

[housefull]

Regarding complaints of COM extraction not working on older releases of InstallShield (mostly pre-IS 12 on XP SP2 or newer machines with COM servers built in VS 2005 or newer), this is due to the API hooking that COM extraction used to rely on failing at a large rate. Since API hooking on modern versions of Windows is not as usable as it once was (possibly because of large sweeping security changes), we decided to use registry redirection with the RegOverridePredefKey API as it yielded a much higher success rate in scenarios where API hooking was no longer working. Another issue was caused in the standalone build since it used to be built on a modified codebase compared to the IDE code (this was eliminated with IS 2009 so both IDE and standalone codebases are identical) that could cause issues specific to standalone build. Registry filtering was added in IS 12 to address the new COM extraction's tendency to pick up unrelated registry keys.
_________________

[CChong]

Sorry, housefull, I don't get it. You cited a paragraph from joshstechnij's post ... and now :confused:
The registry keys I pointed to ARE related, vitally important for the whole multimedia sector!

[joshstechnij]

Thank you for providing this information.

The behavior you are seeing is, as I mentioned in a previous post, likely due to trying to prevent certain keys (such as HKLM\Software) from being included in captured COM data in an MSI package (if this is included, MSI will try to remove all of HKLM\Software on uninstall).

To attempt to avoid this situation, we look at the keys that were captured during extraction. If there are keys that have no subkeys and sub-values, the registry key is removed from the captured output so that it cannot be imported into an ISM/MSI. There are a couple of exceptions to this rule that allow specific empty keys through. Unfortunately, this list of exceptions is currently hard-coded. I have submitted work order IOA-000053506 to address this issue.

On the off chance that the behavior you are seeing is not related to our registry data filtering, would it be possible for you to send us a sample DLL that can reproduce this behavior?

[CChong]

Hi Josh, I can pass one of our demo filters, they are for free anyway. Tomorrow at work, I'm in Europe.
Btw, not that confusion arises: I said "below" those keys (which are never empty), there'll be a new subkey having entries having values...

[CChong]

I attached a DShow filter including dependency .dll; and checked once more (running IS 2009 under Vista64, UAC on) that COM extraction only fetches its 3 GUID subkeys directly below HKCR/CLSID/, i.e. the CLSID and 2 property page tab IDs. Everything below the mentioned ActiveMovie key is missing, FilterData, Capabilities, FriendlyName, well all the stuff what makes up a DShow filter.

I see that your "filters.xml" is well aware of the importance of that {083863F1-} key, it seems to exclude all DShow filters natively shipping with Windows.

Btw, Josh, watch out for quartz.dll from the DirectX distribution; it carries tons of that kind of COM data. And under Windows 7 also for mf.dll - Windows Media Foundation's main library. WMF is the successor of DirectShow and Media Foundation transforms register pretty different again.

[joshstechnij]

We've been able to reproduce the issue you are reporting with the attached files. It appears that filtering is not the cause of this particular issue as an initial look at the behavior shows the registry keys you mentioned are not even being written to our captured information (at least according to Process Monitor).

We'll keep looking at this issue and update this thread when we have more information available.

Thank you for providing these files.

[CChong]

Great! I'll remove my attachment again for privacy reasons assuming that you downloaded and stored it.

[joshstechnij]

After some further investigation on the files that were attached, it appears the behavior occurring here is a result of the registry being remapped with RegOverridePredefKey. Since this effectively remaps all registry accesses to keys we provide, anything that depends upon existing registry data will be unable to access it. In this case, there are class ID keys related to the DirectShow runtime that are not found while registering the .ax file. It is likely that because these keys are not found, the code that registers information specific to the filter is skipped. We have an existing work order open (IOC-000075504) regarding this specific behavior, but at this time no code based solution is available.

Unfortunately, the only work around available at this time would be to use self registration or manually import the registry keys that are missing.

[CChong]

I can live with it, I always had to. 😉
Also because nobody so far could convince me of the big advantage of COM extraction compared to self-registration. Maybe it would handle unregistration on uninstalling a bit better than my own methods (which obey sharedDll protection); but the worst thing with COM server (un)registration even the most sophisticated approach can never completely handle, IMHO:
if Product A registers filter.ax in location 1 and Product B registers the same in location 2, there's nothing in the universe to stop uninstallation of either one to steal the COM server from the other one. Is it?

(Before you answer, consider the scenario that Product B is developed years later, half the globe away, by another license customer not even knowing that Product A ever existed, which nevertheless could be installed before or after B...)

[joshstechnij]

There are pros and cons to each approach to registering COM servers. Most people don't like auto-repair/install on demand but registering COM servers with the Class table (populated through COM extraction) is one way this functionality is provided by MSI. Using self registration will avoid auto-repair or install on demand functionality but it has its own limitations (for example, it doesn't support rollback).

We do want to be able to support extracting any registry through COM extraction, but the way it functions currently limits its ability to capture some data (such as this case with DirectShow information). We hope to address this in a future release (mostly due to the development effort required).