Scanning VMWare Infrastructure

[brianmcelraft]

Hi,

I've recently set up a FNMS 2017 R2 instance and I'm trying to get it to successfully scan a pre-existing VMWare Infrastructure. The Service Account I'm using has Read Only Access to the VCenter Infrastructure. It can succcessfully log in to the VCenter Instance web page. All elements of the VCenter cluster can be pinnged from the FNMS server. The only actual diagnostics I can derive are:

1. Log files from the Inventory Beacon -
2018-01-31 13:58:29,886 [ource.NMapDeviceSource|IPScan] [INFO ] Starting an IP scan.
2018-01-31 13:58:29,933 [iscovery.NMapDiscovery|IPScan] [INFO ] Building command line with TCP ports:22,80,135,139,161,443,445 and UDP ports 22,80,135,139,161,443,445
2018-01-31 13:58:29,933 [iscovery.NMapDiscovery|IPScan] [INFO ] Command line for mgsipScan: -p T:22,80,135,139,161,443,445,U:22,80,135,139,161,443,445 -oX "C:\Users\flx-fnms-svc\AppData\Local\Temp\ManageSoft\discovery\mgsipscan-t-2018131_135829-64fe98833c-07fa-4ce3-b6f2-bdc9e6ef6926.xml" -PI -sS -sU 10.0.0.10
2018-01-31 13:58:30,105 [veryExportDeviceSource|DeviceSource] [INFO ] Processing exported disco files extracted in the folder 'C:\ProgramData\Flexera Software\Beacon\DiscoveryExport\19': 1 files to be processed.
2018-01-31 13:58:30,105 [veryExportDeviceSource|DeviceSource] [INFO ] Processing exported disco file 'C:\ProgramData\Flexera Software\Beacon\DiscoveryExport\19\1.disco'
2018-01-31 13:58:30,309 [iscoveryActionExecuter|DNSLookup] [INFO ] Failed to perform DNS discovery for device '10.0.0.10'.
2018-01-31 13:58:32,234 [iscoveryActionExecuter|Async] [INFO ] Enumerating devices from store after DNS lookup
2018-01-31 13:58:32,234 [iscoveryActionExecuter|Async] [INFO ] Completed enumeration of devices from store after DNS lookup
2018-01-31 13:58:36,914 [iscoveryActionExecuter|IPScan] [INFO ] Device '10.0.0.10' is in scope: VCenter
2018-01-31 13:58:36,914 [ource.NMapDeviceSource|IPScan] [INFO ] Completed an IP scan.
2018-01-31 13:58:36,946 [.DiscoveryTaskExecutor|PropertyDisco] [INFO ] Performing Device Property Scan discovery for device '10.0.0.10'
2018-01-31 13:58:36,946 [veryDevicePropertyScan|PropertyDisco] [INFO ] Performing nbtstat discovery for device: 10.0.0.10
2018-01-31 13:58:37,029 [veryDevicePropertyScan|PropertyDisco] [INFO ] Performing ARP discovery for device: 10.0.0.10
2018-01-31 13:58:37,042 [.MacAddressScanInterop|PropertyDisco] [INFO ] The specified IPAddress '10.0.0.10' is not within the same subnet.
2018-01-31 13:58:37,042 [veryDevicePropertyScan|PropertyDisco] [INFO ] Performing NetAPI discovery for device: 10.0.0.10
2018-01-31 13:58:38,259 [iscoveryActionExecuter|Async] [INFO ] Starting to enumerate devices from IP Scan
2018-01-31 13:58:38,259 [iscoveryActionExecuter|Async] [INFO ] IP Scan completed, 0 devices failed to get port info. Adding these devices to downstream.
2018-01-31 13:58:59,113 [overy.NetServerGetInfo|PropertyDisco] [INFO ] Completed NetAPI discovery for device '10.0.0.10' unsuccessfully: The network path could not be found
2018-01-31 13:58:59,114 [.DiscoveryTaskExecutor|PropertyDisco] [INFO ] Completed Device Property Scan discovery for device '10.0.0.10': DeviceName = "", MAC Address = "", NTDomain = "" and WindowsType = 0.
2018-01-31 13:58:59,114 [.DiscoveryTaskExecutor|PropertyDisco] [INFO ] Performing VMware discovery for device '10.0.0.10'
2018-01-31 13:58:59,151 [.DiscoveryTaskExecutor|PropertyDisco] [INFO ] Completed VMware discovery for device '10.0.0.10': not discovered
2018-01-31 13:59:00,337 [iscoveryActionExecuter|Async] [INFO ] Discovery execution has completed.
2018-01-31 13:59:00,347 [coveryThreadedExecutor|Async] [INFO ] Shutdown of 10 threads for device discovery.
2018-01-31 13:59:00,352 [coveryThreadedExecutor|Async] [INFO ] Shutdown of 10 threads for DNS lookup.
2018-01-31 13:59:00,352 [coveryThreadedExecutor|Async] [INFO ] Shutdown of 1 threads for network scanning.
2018-01-31 13:59:00,352 [coveryThreadedExecutor|Async] [INFO ] Shutdown of 20 threads for device property and service discovery.

and a Wireshark log which shows:
a 301 response code from the FNMS server posting to https:///sdk.

Does anyone have any ideas?

[ChrisG]

Hi brianmcelraft,

Some common problems would be that the VMware web API is not running on 10.0.0.10, or that the beacon does not have network connectivity to port 443 (HTTPS) on the VMware server.

For troubleshooting, you could try using a web browser running on the beacon to browse to https://10.0.0.10/mob and ensure you do not get a "connection failed" or "404 not found" type response. (You may still get an error browsing to that URL, but you want an error which indicates that a connection was successfully made, and that there seems to be something running there.)

Chris @ Flexera

[brianmcelraft]

cgrinton wrote:
Hi brianmcelraft,

Some common problems would be that the VMware web API is not running on 10.0.0.10, or that the beacon does not have network connectivity to port 443 (HTTPS) on the VMware server.

For troubleshooting, you could try using a web browser running on the beacon to browse to https://10.0.0.10/mob and ensure you do not get a "connection failed" or "404 not found" type response. (You may still get an error browsing to that URL, but you want an error which indicates that a connection was successfully made, and that there seems to be something running there.)

Chris @ Flexera

Unfortunately I've tried both of these. The only clue I can gander from the logs is that the Beacon is having issues connecting to https:///sdk/. The actual REST interface is sitting at https:///sdk/vim.wsdl. I have no issues connecting to it from the browser level on the beacon itself via HTTPS...

[JohnSorensenDK]

brianmcelraft wrote:
Unfortunately I've tried both of these. The only clue I can gander from the logs is that the Beacon is having issues connecting to https:///sdk/. The actual REST interface is sitting at https:///sdk/vim.wsdl. I have no issues connecting to it from the browser level on the beacon itself via HTTPS...

You may want to test/troubleshoot using the ESXQuery inventory tool that can be executed manually, please download it from https://flexeracommunity.force.com/customer/articles/en_US/INFO/Additional-Inventory-Tools-for-FlexNet-Manager-Suite

Thanks,
John Sorensen
Flexera

[brianmcelraft]

JohnSorensenDK wrote:
You may want to test/troubleshoot using the ESXQuery inventory tool that can be executed manually, please download it from https://flexeracommunity.force.com/customer/articles/en_US/INFO/Additional-Inventory-Tools-for-FlexNet-Manager-Suite

Thanks,
John Sorensen
Flexera

Thanks for the reply John. I've tried to run ESXQuery on a production instance where similar issues are being seen. In both cases it looks like the ESXuery (and FNMS in general) are trying to query https://:443/sdk. However, this url looks to be deprecated by VMWare.
Here's the relevant output from ESXQuery
BindServer(, proto=https, port=0) failed.
In fsend call to WinHttpSendRequest: A connection with the server could not be established (12029)
An error occured in HTTP processing
Failed to retrieve contents from web service https://:443/sdk
BindServer(.com, proto=https, port=0) failed.

I'm able to manually browse to the VCenter console from both machines. One issue is that we're using self signed certificates in these cases. However, at least in my test environment, I've tried to loosen up Cert requirements to deal with this.

Any ideas?

Thanks

Brian

[ChrisG]

The "A connection with the server could not be established" error certainly suggests a network connectivity sort of problem (as opposed to anything to do at the application software layer).

A couple more thoughts:

[LIST=1]

What version of VMware software is running on the server? Check that it is a version that is supported by the FlexNet Manager Suite release you are using, including that it is not a free VMware ESXi install (which doesn't expose the APIs required to gather the information needed here).

Is it possible that the web browser is configured to go through a web proxy to reach the VMware server? The beacon & ESXQuery tool will likely not be using any web proxy, which could explain the browser being able to connect but the beacon failing to connect.


Chris @ Flexera

[brianmcelraft]

I've verified connectivity from the FNMS Beacon via the ESXQuery tool. In addition, I've loosened up some ACLs for the ports (80 and 443) listed as necessary for the VMWare scanning components to work. The rule fires and completes, however in the actual Inventory phase, the only information provided by the beacon is the following log entry -
2018-02-13 09:57:46,360 [.VirtualizationVisitor|Async] [INFO ] No inventory gathered for device '' because discovery did not find the service on the device.

The ESXQuery is able to download an inventory without issue.

[Jennifer_Munson]

We've had the same issue with various vCenters. More and more are being upgraded to vCenter 6.5, and then we can't use the esxquery script as a workaround any more.

Additional troubleshooting steps Flexera technical support has tried:

- Ensure the Discovered Device exists; if not create it
- Ensure the Discovered Device has the correct DNS domain and FQDN that can be discovered from the beacon
- Set up the Target to use the Discovered Device name instead of IP address
(wait... sometimes this takes a day or two to work)

- Change mgsipScan options using registry keys on the beacon <- would recommend opening a case with Flexera to get specifics on how to do this

I'll be interested to hear how you get yours solved!

[qassnovo]

Found this conversation in Google and had exactly the same problem. In our case it was caused by Windows and its TLS 1.2 support.

I was able to sort it out for ESXQuery. Unfortunately the Beacon still won't talk to the service but ESXQuery is a good workaround. 

On the Beacon open registry key: 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp

Create a DWord entry called DefaultSecureProtocols

- To support TLS 1.1 AND 1.2 assign the following hex value: A00

- To support TLS 1.2 only assign the following hex value: 200

- To support TLS 1.1 only assign the following hex value: 800

Just to be on the safe side I also created the same entry in 

HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp

We used value A00 and it resolved the probem. Here's a description from Microsoft: https://support.microsoft.com/en-ca/help/4467770/update-to-enable-tls-1-1-and-tls-1-2-as-secure-protocols-on-winhttp

Regards,

A. Starosta