caCreateVRoots causing rollback on Server2008 (IIS7)

[rgreen]

I seen numerous posts on this but most involve applying Hotfix Q200236 for IIS7 which has been on our ISStandaloneBuild on our build server and my development system for quite some time (also have Hotfix Q113652). Those hotfixes did correct the rollback issues that we initially had with IIS 7 on Vista.

We've since dropped Vista support of our product but still have to support Server2008. We've been installing successfully on Server2008 for a few months now, but recently, rollbacks started happening during the caCreateVRoots custom action. On some of our test systems, we've removed the IIS role, reinstalled IIS and sometimes can can get it to install once again. But after the first uninstall/reinstall of the same or another build it starts rolling back again. Once this behavior begins, any prior builds that we've successfully installed will no longer install with the same rollback problem! It seems that an initial install/uninstall of our product changes the system in such a way that no future installs work.

Only recent change that we've recently made is allowing the user to pick the website of their choice to install a virtual directory w/ a default name that they can edit. To do that, I wrote installscript CA's that gather website info using the ADSI or WMI api's depending if we're on IIS6 or IIS7.
We're pushing to get a Release Candidate out this week but this issue is becoming a show stopper!

I've rambled enough in this post. Attached is a minimalist "Hello World" version of our install that interacts with IIS the same as our production product. It installs 1 simple html file in a virtual dir under a website of the users choosing. True to form, it installed successfully once on 2008 (maybe twice) but then after that, the roll backs start happening. Log attached.
Any thoughts!?

[rgreen]

We're not digitally signing our production install. Would that be relevant?
Could that be why we have to instruct users on 2008 to alt-click, "run as administrator" on our setup.exe (our production version has a setup.exe to install .NET and other 3rd party prereq's) to guarantee that our website enumeration code will work?

[rgreen]

We've been avoiding requiring the IIS6 metabase compatibility mode on IIS7. But as we're getting more and more desperate to meet our schedule may be we'll have to back off that requirement.

[rgreen]

Gets stranger all the time. Now after uninstall/reinstalling IIS on our test 2008 server. That simple install that was failing is now working. Not too, surprising, seen that happen before, but it continues to work after 4 install/uninstall cycles. In the middle of those, I installed our production install and it failed right off the bat. Attached is it's log, search for caCreateVRoots further down in it. Sorry, it's quite large.

The problem section that initiates the rollback:

1: put_value succeeded. get_StringValue now returns '0' with string value 'Read, Execute, Script'. 
1: CommitChanges for path 'MACHINE/WEBROOT/APPHOST/Default Web Site/TracerES' and physical path 'C:\Program Files\Trane\TracerES\UI\'. 
1: CommitChanges failed with the following HRESULT: '-2147024891'.  
1: Error with IISRT: -2147024891 
Action ended 0:02:44: InstallExecute. Return value 3.

[joshstechnij]

The last error (-2147024891) indicates access is denied when attempting to modify the IIS configuration data. Is there a web.config file present in the physical directory your virtual directory is pointing to? If so, please ensure this file is not being installed read-only and has sufficient permissions for the SYSTEM account and current user to be able to modify the file.

[rgreen]

joshstechnij wrote:
The last error (-2147024891) indicates access is denied when attempting to modify the IIS configuration data. Is there a web.config file present in the physical directory your virtual directory is pointing to? If so, please ensure this file is not being installed read-only and has sufficient permissions for the SYSTEM account and current user to be able to modify the file.

Josh, Thanks for the prompt reply!
Yes we do have a web.config in our web application. And no we haven't explicitly set R/W permissions on that file. I'll look into doing that!
What is odd is that ISXmlInstall is running just before caCreateVRoots were we are doing some IS Xml Changes to web.config, and that appears to run OK. Do you know what user we should grant access to? Early on we set 3 IIS related username properties, on 2008/IIS7 this are getting set to:
[IISUSER_IWAM] = IUSR
[IISUSER_IUSR] = IIS_IUSRS
[IISUSER_ASPNET] = IUSR

After caCreateVRoots we invoke (via LaunchAppAndWait) CACLS.EXE from deferred installscript CA's that set full privileges to the [IISUSER_ASPNET] user on various sub folders under the physical path of our VDir. Should we move all that before caCreateVRoots (or ISXmlInstall to be safe) and add web.config to that list?

Anyway, that's what I'm going to try. Thanks again!

[rgreen]

rgreen wrote:
... Do you know what user we should grant access to? ...

Sorry I missed your comment to grant the SYSTEM user. And for the current user, I see a property [LogonUser] that is getting set. Assuming that is what I should go with for current user?
Thanks again.

[joshstechnij]

Yes, SYSTEM should have access and the user that launched the install (LogonUser should work).

Note that another possible cause of this behavior could be the contents of the web.config file. As of now, we are aware of a limitation with IIS 7 on Vista SP1 and Windows Server 2008 that will prevent IIS from successfully updating web.config for an application/virtual directory if that web.config uses XML namespaces (see http://forums.iis.net/p/1147731/1862643.aspx). This appears to be resolved with IIS 7 on the beta versions of Windows 7 and Windows Server 2008 R2. If permissions do not seem to be the issue, check the contents of the file to see if namespaces are used.

[rgreen]

joshstechnij wrote:
Yes, SYSTEM should have access and the user that launched the install (LogonUser should work).

Note that another possible cause of this behavior could be the contents of the web.config file. As of now, we are aware of a limitation with IIS 7 on Vista SP1 and Windows Server 2008 that will prevent IIS from successfully updating web.config for an application/virtual directory if that web.config uses XML namespaces (see http://forums.iis.net/p/1147731/1862643.aspx). This appears to be resolved with IIS 7 on the beta versions of Windows 7 and Windows Server 2008 R2. If permissions do not seem to be the issue, check the contents of the file to see if namespaces are used.

Josh,
Still working on testing out the permissions test. However, I'm getting a bit pessimistic on that solution but I'll still give it a try. I discovered old folders created by our original installs (folder creation dates are in the 11/08 timeframe) on that 2008 test system that installed fine last year. Subfolders created by our product at runtime caused the uninstalls to leave those behind. So the remaining web app root folder left behind did have full access assigned to SYSTEM and the Administrators group (person running the install is in that group). I'm guessing that is how they were originally set by the install.

Another thing that I tried is installing to some place
outside of 'c:\program files' in this case, 'C:\Users\[myuser]\[productname]'
and it installed fine. Full rights for SYSTEM and Admin users appear to inherit down that path.

As far as our web config goes, this is it's top level structure.

So yes it does use namespaces. I'll look at that forums.iis.net post thanks.

However, the fact that we've actually installed on our 2008 server and ran the product fine on the not too distant past would see to negate these theories. It was on a specific date on this server that the VDir rollbacks first happened. I'll should look back if the windows event log goes back that far (3 weeks or so) and if a windows update was pushed to that server by our test network admin.

[joshstechnij]

I would recommend the following approach to try isolating the behavior:
- Install a version of the product that does not install the web.config file (and also does not modify it with XML file changes)
- If the above works, add a blank web.config to the install and test again
- If the above works, start adding elements back to the file and test again until a failure occurs or all elements are back in the file
- Re-add the file to XML file changes if the above succeeded with all elements restored to the file

If none of the above resolve the behavior, the next thing to look at could be the applicationHost.config file maintained by IIS on the machine. It is possible that this file has gotten into a state that is prevent IIS from updating certain settings correctly.

[rgreen]

Very strange behaviors being discovered now. Not sure whether to be frustrated or entertained!
So I started on the suggested testing path:

joshstechnij wrote:
I would recommend the following approach to try isolating the behavior:
- Install a version of the product that does not install the web.config file (and also does not modify it with XML file changes)

This failed with the same rollback. OK, so I think, well perhaps something about my Vdir definition is telling the CA or an MS api the CA is calling to expect a web.config where there is now none and trips it up. So I move on...

joshstechnij wrote:
- If the above works, add a blank web.config to the install and test again

Rolls back again. Hmm OK, just to be sure the blank web.config is getting installed, I temporarily condition the caCreateVRoots CA to false so the install works and I can see what is being installed. Hmmm, no web.config file in sight. I run the install on my development PC (running XP). The blank web.config is installed as expected. So I search the log for references to my component that holds the web.config file 'UI.WebConfig'. On my XP system, in InstallValidate I see this component is scheduled for installation:

MSI (s) (58:58) [23:48:03:567]: Doing action: InstallValidate
Action 23:48:03: InstallValidate. Validating install
Action start 23:48:03: InstallValidate.
MSI (s) (58:58) [23:48:03:567]: Feature: UI; Installed: Absent; Request: Local; Action: Local
MSI (s) (58:58) [23:48:03:567]: Component: UI.WebConfig; Installed: Absent; Request: Local; Action: Local

However on my Server2008 system, this is InstallValidate:

MSI (s) (30:0C) [23:55:23:982]: Doing action: InstallValidate
Action 23:55:23: InstallValidate. Validating install
Action start 23:55:23: InstallValidate.
MSI (s) (30:0C) [23:55:23:998]: PROPERTY CHANGE: Deleting MsiRestartManagerSessionKey property. Its current value is '33e72f3fe5313c49b24d1bf56f2bd074'.
MSI (s) (30:0C) [23:55:23:998]: Feature: UI; Installed: Absent; Request: Local; Action: Local
MSI (s) (30:0C) [23:55:24:013]: Component: UI.WebConfig; Installed: Absent; Request: Local; Action: Null

What! it's like Server2008's MSI engine is saying "I'm just not going to install web.config and that's final!" So I searched other failed install logs from earlier failed installs and they all show the same thing, web.config is simply not getting scheduled for installation! The other question then is, what is caCreateVRoots getting tripped up by if web.config went AWOL in the first place?

Note that on that one successful install that I did outside of of c:\program files, 'web.config' did get scheduled for installation in InstallValidate:

MSI (s) (B4:18) [13:21:27:057]: Component: UI.WebConfig; Installed: Absent; Request: Local; Action: Local

and that install worked. However the install location I picked, c:\users\[myuser]\[ourproduct], tripped up IIS accessing web.config at runtime. I'm going to see if installing our product under IIS's default location of: c:\inetpub\wwwroot would fair any better?

Oh, I should mention, that I suspected UAC being a culprit last week, so I temporarily disabled it to eliminate that possibility. OK, too much excitement for one day. I'll have to pick this up back in the office.
All for now...

[rgreen]

Our theory is that Microsoft via a recent windows update that we only recently picked up is now forbiding IIS7 web applications to install under "Program Files". Perhaps they are trying to avoid a security risk of a website being comprised and the hacker getting access to content under program files?

This was our existing layout:
[IISROOTFOLDER] <-- we didn't touch this
[ProgramFilesFolder]

[Company]

[Product] <-- [INSTALLDIR] User can change this in UI

Server <-- our backend server running as a windows service

bin

logs

UI <- Our IIS VDir gets pointed here

Any this is what we're going to try to fix our Server2008 install:
[IISROOTFOLDER]

[Product] <- Our IIS VDir gets pointed here

[ProgramFilesFolder]

[Company]

[Product] <-- [INSTALLDIR] User can change this in UI

Server <-- our backend server running as a windows service

bin

logs

[rgreen]

Update, fails even when installing to [IISROOTFOLDER]\[Product] !
Web.config is now getting installed but the install still rolls back. I ran a version that skips caCreateVRoots to verify that r/w permissions are set on web.config and they are.

I'll re-try those other troubleshooting steps that Josh recommended. However, currently, to cut our loses, I'm putting in an "IIS rollback safety net" that skips caCreateVRoots when the property: IIS_BYPASS=1 is passed on the command line. We'll then provide a batch file on our CD media to launch our install with this property getting set.

In some alternate screens in the UI, we'll direct them to our "Installation Guide" for instructions to manually create the virtual directory.

Our rollback when installing to [IISROOTFOLDER]\[Product]:

MSI (s) (20:4C) [13:10:39:316]: Executing op: ActionStart(Name=caCreateVRoots,Description=Creating IIS Virtual Roots...,)
Action 13:10:39: caCreateVRoots. Creating IIS Virtual Roots...
MSI (s) (20:4C) [13:10:39:331]: Executing op: CustomActionSchedule(Action=caCreateVRoots,ActionType=3073,Source=BinaryData,Target=CreateIISVRoots,CustomActionData=C:\Users\rgreen\AppData\Local\Temp\IISB1E4.tmp)
MSI (s) (20:A0) [13:10:39:347]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIB4FA.tmp, Entrypoint: CreateIISVRoots
1: This machine does not support legacy objects
1: InitIISObject
1: CoGetClassObject(GETAdminBaseCLSID(TRUE) error: -2147221164
1: Legacy objects not supported
Action 13:10:39: Progress Custom Action. Creating application Pools...
Action 13:10:39: Progress Custom Action. Creating web service extensions...
Action 13:10:39: Progress Custom Action. Creating IIS virtual directories...
Action 13:10:39: Progress Custom Action. Creating IIS virtual directory TracerES
1: SetCommitPath for website '' and app ''
1: InstallShield project contains a website with name 'Default Web Site', site number '1', and port '80'. Checking if it exists...
1: GetAdminSection for section 'system.applicationHost/sites' and path 'MACHINE/WEBROOT/APPHOST' and commitpath 'MACHINE/WEBROOT/APPHOST'.
1: GetAdminSection returned: -2147024883. The data is invalid.
1: Error: already using the main configuration file.
1: No sites section found
1: Website does not exist. Additonal info: Site number '1', Port '80', sBasePath '/LM/W3SVC/', DisplayName 'Default Web Site'
1: GetAdminSection for section 'system.applicationHost/sites' and path 'MACHINE/WEBROOT/APPHOST' and commitpath 'MACHINE/WEBROOT/APPHOST'.
1: GetAdminSection returned: -2147024883. The data is invalid.
1: Error: already using the main configuration file.
1: Element is NULL. Can't get collection
1: Collection is NULL. CreateNewElement failed for: site
1: PutPropertyValue: sProperty name sValue Default Web Site bIsPath 0
1: PutPropertyCommon: sProperty name sValue Default Web Site bIsPath 0
1: Element is NULL while looking for the following property: name
1: PutPropertyCommon: no property returned for name, don't try to set it now
1: PutPropertyValue: sProperty id sValue 1 bIsPath 0
1: PutPropertyCommon: sProperty id sValue 1 bIsPath 0
1: Element is NULL while looking for the following property: id
1: PutPropertyCommon: no property returned for id, don't try to set it now
1: GetBindingsCollection
1: pParentElement NULL while looking for sub 'bindings'
1: Element is NULL. Can't get collection
1: Collection is NULL. CreateNewElement failed for: binding
1: GetBindingInformationString is returning the following string: *:80:
1: PutPropertyValue: sProperty bindingInformation sValue *:80: bIsPath 0
1: PutPropertyCommon: sProperty bindingInformation sValue *:80: bIsPath 0
1: Element is NULL while looking for the following property: bindingInformation
1: PutPropertyCommon: no property returned for bindingInformation, don't try to set it now
1: PutPropertyValue: sProperty protocol sValue http bIsPath 0
1: PutPropertyCommon: sProperty protocol sValue http bIsPath 0
1: Element is NULL while looking for the following property: protocol
1: PutPropertyCommon: no property returned for protocol, don't try to set it now
1: spCollection NULL in CIisFunctionsVista::AddElement.
1: pElement NULL in CIisFunctionsVista::AddElement
1: Exception caught in CreateVRoot function
1: Error with IISRT: -2105
Action ended 13:10:39: InstallExecute. Return value 3.
MSI (s) (20:4C) [13:10:39:831]: User policy value 'DisableRollback' is 0
MSI (s) (20:4C) [13:10:39:831]: Machine policy value 'DisableRollback' is 0
MSI (s) (20:4C) [13:10:39:863]: Executing op: Header(Signature=1397708873,Version=400,Timestamp=979724597,LangId=1033,Platform=0,ScriptType=2,ScriptMajorVersion=21,ScriptMinorVersion=4,ScriptAttributes=1)
MSI (s) (20:4C) [13:10:39:863]: Executing op: DialogInfo(Type=0,Argument=1033)
MSI (s) (20:4C) [13:10:39:863]: Executing op: DialogInfo(Type=1,Argument=Tracer ES 2.0 Server Edition)
MSI (s) (20:4C) [13:10:39:863]: Executing op: RollbackInfo(,RollbackAction=Rollback,RollbackDescription=Rolling back action:,RollbackTemplate=[1],CleanupAction=RollbackCleanup,CleanupDescription=Removing backup files,CleanupTemplate=File: [1])
Action 13:10:39: Rollback. Rolling back action:
Rollback: Creating IIS Virtual Roots...

[rgreen]

Noticed something in the event log on our 2008 test server.
These are filling our windows application event log on 10 minute intervals. would these account for the difficults of caCreateVRoots running?

Log Name: Application
Source: Microsoft-Windows-IIS-W3SVC-WP
Date: 3/5/2009 2:49:45 PM
Event ID: 2297
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: twctesweb08
Description:
The worker process for application pool 'DefaultAppPool' encountered an error 'Configuration file is not well-formed XML
' trying to read global module configuration data from file '\\?\C:\inetpub\temp\apppools\DefaultAppPool.config', line number '3'. Worker process startup aborted.
Event Xml:



2297
0
2
0
0
0x80000000000000

6689


Application
twctesweb08



DefaultAppPool
Configuration file is not well-formed XML

\\?\C:\inetpub\temp\apppools\DefaultAppPool.config
3
0D000000

[joshstechnij]

That event log information could explain the install failures you are seeing. If you can't change any settings through the IIS manager on your application, this could be related. The file C:\inetpub\temp\apppools\DefaultAppPool.config would either need to be corrected or removed to resolve this.

Another possible cause of the behavior you are seeing is left over information in C:\Windows\System32\inetsrv\config\applicationHost.config related to your application. Uninstalling/reinstalling IIS should fix this for one install, and then all subsequent uninstalls/reinstalls of your product could potentially fail. You might try the following to see what is changing between a fresh install of IIS and installing/uninstalling your product:
- Uninstall and then reinstall IIS
- Copy C:\Windows\System32\inetsrv\config\applicationHost.config to some safe location
- Install and then uninstall your project
- Compare C:\Windows\System32\inetsrv\config\applicationHost.config with the copy saved earlier with a text/xml diff utility to see if there are any changes left over

[rgreen]

Thanks for the reply Josh. We found the source of the problem today, and I'm a bit embarrassed to report that it was from a bug I introduced about 3 weeks ago. But there are a few lessons here that I'm happy to share with anyone that reads this thread!

Your instincts about a problem with applicationHost.config were spot on. An issue we identified on some Vista PCs a while back was an attribute for controlling static content was set to false (in spite of the "static content" feature being checked in Vista's IIS features UI). Even though Server2008 is now the only IIS7 platform we support, we figured we should be sure that feature is always on because it's rather hard to deliver our web content without IIS serving our graphic files! So I added an XmlChange item to force it true on IIS7 platforms. I was a bit uncomfortable modifying such an important system file, but I thought I'd gotten the nack of XmlChanges. XmlChanges for us so far were on our own installed .NET config files. Well, I got sloppy (late in the day too) and didn't scrutinize all the default settings on all the element nodes above the attribute we were modifying. You probably know that you default new XmlChange element records to remove on uninstall. OK, you can probably imagine what happened next! With most elements having this set, every uninstall would now remove a huge chunk of applicationHost.config out from under IIS!

Most of our testing focuses on IIS6 and we didn't seriously turn our attention to this until last week. In a conversion today with someone on the project, they said that some of the testers were starting to suspect IIS corruption. That jogged my memory of that "small" fix to the IIS config file from a while back, so I immediately went to my desk and in horror found all these delete on uninstall flags set! I've cleared them for our own config files but wasn't in the habit of being too concerned with them since we delete all our own config files on uninstall anyway.

So with IIS7's configuration being destroyed on every uninstall, the install of the next build would experience whatever side effects the devastation to IIS created by the preceding uninstall would cause (including tripping up your caCreateVRoots code)! Knowing what I now know of what was happening, I find some of the manifested symptoms interesting, like MSI not installing our web.config under 'program files' but elsewhere. It's sinking in how much MS infrastructure and API's are probably depending on these config files and thus all the wild and weird behaviors that result when those files are not handled with care.

Some of our testers were recovering by reinstalling IIS (role remove/add). I'd removed/re-added the IIS role several times but that didn't fix me until today. Feels like the remove/re-add IIS role thing isn't the same as restoring to a pristine state and maybe I'd damaged my IIS enough that it took serveral tries to fully get it back? Dunno?

OK, so I claim all responsibility for the XmlChange fiasco, wasn't "driving with my eyes open". However, would a reasonable suggestion be for safer default settings in XmlChanges? It's sorta like a loaded gun when the file pointed at isn't your own. At least determine if a modified file is outside of the [INSTALLDIR] structure and thus possibly being an existing system file. Then, in that case, default all XML deletion settings to off? Maybe something to pass on to the XML change people?

Anyway thanks again for your guidance, Josh!