Cannot discover / inventory VMware vSphere 5 from Windows Server 2003

[#1]

Summary

Failure to discover / inventory VMware vSphere 5 vCenter or ESXi server from a Windows Server 2003 or Windows XP distribution server (or administration server).

Symptoms

Failure to discover / inventory VMware vSphere 5 vCenter or ESXi server from a Windows Server 2003 or Windows XP distribution server (or administration server)

A HTTPS connection to vCenter 5 or ESXi 5 server could not be established. To confirm that this is the reason for failure, enable tracing of GSoap (+GSoap in etap.trace) and look for the error message:

In fsend call to WinHttpSendRequest: The server returned an invalid or unrecognized response (12152)

Cause

This is caused by an incompatibility between the TLS implementations in vSphere 5 servers and the Windows XP / Server 2003 WinHTTP client library used by the FNMP discovery and inventory agents. A combination of two changes made by VMware in vSphere 5 has the result that there is no TLS cipher supported in common by the client and server in this scenario by default.

Before vSphere 5, both RC4 and 3DES ciphers were supported by these clients and servers. In vSphere 5, its configuration was changed to remove RC4 from the default list of supported ciphers. Also, it implements a countermeasure to the 'BEAST' attack on TLS 1.0 that WinHTTP on the listed operating systems cannot handle, which involves sending an initial empty packet when using CBC ciphers including 3DES.

Resolution

Change the SSL/TLS configuration for affected vCenter 5 servers to add support for RC4 ciphers. This is done by adding a <cipherList> element inside the <config> <vmacore> <ssl> section as follows into the server-dependent configuration XML file and then restarting the VMware vCenter Server service or ESXi Management Agents.

vCenter 5 servers (Windows 2003 and below): C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg
vCenter 5 servers (Windows 2008 and above): C:\ProgramData\VMware\VMware VirtualCenter\vpxd.conf
ESXi 5 servers: /etc/vmware/hostd/config.xml

<config>
<vmacore>
<ssl>
<cipherList> "HIGH:RC4-SHA:RC4-MD5" </cipherList>
</ssl>
</vmacore>
</config>

Note that not all security experts would agree with the description of RC4 as a 'nonsecure' cipher in the first VMware KB article below - please obtain your own advice about the level of security required in your environment before making this change.

Workaround

If the above configuration changes cannot be made to vSphere 5 servers, the workaround is to perform the discovery / inventory task on a distribution server running a later operating system, for example Windows Server 2008.

Additional Information

ESXi 5.0 disables nonsecure ciphers in Internet Explorer 6

VirtualCenter Server 2.5 Update 4 and later uses high-security SSL ciphers

Ciphers supported on ESX and vCenter Server

Restarting the Management agents on an ESX or ESXi Server