FNMS agent installed on Windows server 2003 and 2008 were not reflecting

[winvarma]

Hello Forum,

We are in the phase of initial setup of FNMS cloud 2019 R1.3 and have deployed agents(13.6.0) on few windows test servers (2003, 2008, 2012, 2016 & 2019) OS versions.

The FNMS agents  below 2012 OS were not reflecting in the Suite. Identified that the agents were giving the below error in the installation log

**********************************************************************************************************

[6/25/2019 4:05:30 PM (G, 1)] {4264} Base URL “https://xyz.com/ManageSoftDL/” will be used
[6/25/2019 4:05:30 PM (N, 0)] {4264} Downloading “https://xyz.com/ManageSoftDL/Policies/Merged/xyz.com_domain/Machine/xyz.npl?machinename=xyz&ipaddress=XXX.XXX.XXX.XXX” to “C:\Windows\TEMP\NDL9225.npl”
[6/25/2019 4:05:30 PM (G, 0)] {4264} Download failure: An existing connection was forcibly closed by the remote host.
[6/25/2019 4:05:30 PM (N, 0)] {4264} Download FAILED for “https://xyz.com/ManageSoftDL/Policies/Merged/xyz.com_domain/Machine/axcndevdb01.npl?machinename=axcndevdb01&ipaddress=XXX.XXX.XXX.XXX”
[6/25/2019 4:05:30 PM (N, 0)] {4264} Downloading “https:/xyz.com/ManageSoftDL/Policies/Merged/xyz_domain/Machine/xyz.npl?machinename=axcndevdb01&ipaddress=XXX.XXX.XXX.XXX” to “C:\Windows\TEMP\NDL9225.npl”
[6/25/2019 4:05:30 PM (G, 0)] {4264} Download failure: An existing connection was forcibly closed by the remote host.
[6/25/2019 4:05:32 PM (U, 0)] {4264} ERROR: Error (s107m858)
[6/25/2019 4:05:32 PM (U, 0)] {4264} ----------------
[6/25/2019 4:05:32 PM (U, 0)] {4264} The following network error occurred while retrieving the application:

An existing connection was forcibly closed by the remote host.

Contact your network administrator for assistance.

*************************************************************************************************************

Suspecting issue might be with the TLS protocol and request you to suggest the work around like registry settings in the client server, as we have only 1 Beacon setup in the solution and how to force the agent to communicate with the Beacon server.

Will be thankful for any leads and suggestions.

Thanks,

Winvarma

[bmaudlin]

This is typically the behaviour experienced in either:

Necessary firewalling hasn't been completed from Agent to Inventory server

Or more likely as its Windows 2012 and lower TLS 1.1/.2 has not been enabled on the servers to be inventoried by the agent. 

As on 2008R2 it is switched off by default, on lower so 2008 or 2003 it cannot be enabled. 

[bmaudlin]

In terms of workaround - there is a number of options - and depending what the environment stipulates is also a factor:

Switch on TLS 1.1/.2 for 2008R2 Servers - however this will not solve the issue on 2003/2008.

Switch TLS off on the inventory beacon for client connections - im guessing here there is at least two beacons (inventory and master)

Enable a second beacon that does not have the requirement for TLS - however the agent configuration will have to differ for these versions 

Decomm the older servers - remove the problem that way 🙂

[winvarma]

hi @bmaudlin ,

Thanks for sharing your thoughts on the issue,but there is no chance to deploy a second Beacon server and  will not be able to switch off TLS as the cloud solution inventory beacon should upload and download data via TLS 1.1/1.2 only due to security reasons(Data packets received by the Flexera Cloud that are encrypted using TLS 1.0 will not be accepted- by Flexera).

The solution is provided for a customer and he might not be a thought on decommissioning the old servers. 

Trying to figure out all the Possible work arounds,

[bmaudlin]

Hi @winvarma 

So I'm guessing that there is not two beacons in place? As my understanding is in terms of the Cloud solution at least it is recommended for at least two beacons to be in place:

So one that connects to the Flexera cloud with the necessary TLS requirement in place, and a second beacon to which collects the inventory data from the agent which doesn't require TLS which sends its data to the beacon that connects to the FLexera cloud.

Ben

[winvarma]

Hi @bmaudlin  yes there is only 1 Beacon and customer didn't accept for 2 beacons may be we will place one one more once the initial phase is completed, and then have to check as suggested.

Thanks for the reply

[craig_moore]

We are seeing this error with Windows 10 devices.  Most are reporting in fine with no errors but some have the error in the installation.log:

12/1/2021 11:22:19 AM (G, 0)] {5076} Download failure: An existing connection was forcibly closed by the remote host.
[12/1/2021 11:22:19 AM (U, 0)] {5076} ERROR: Error (s107m858)

We are thinking maybe related to TLS protocol but do not see a difference in TLS configuration b/w windows 10 that works and one that does not.

Any idea on how to troubleshoot to eliminate this error?  Can we make an update on the beacon server to not close the connection?  You mentioned switching TLS off on the inventory beacon (we only have one in this environment) but not sure how to do this in order to test if resolves our issue.

Thanks

Craig

[winvarma]

Hi @craig_moore ,

try enabling TLS 1.1/1.2 at the client end and see if that resolves the issue and what version is TLS is enabled at Beacon level?

Regards,

[craig_moore]

We have confirmed that TLS 1.1/1.2 at the client is enabled on the Windows 10 device that reports in ok and the device that does not report in.

Checking on the TLS version enabled on the beacon.

[durgeshsingh]

Hi

Are you facing problem in Windows 10 Device or server 2k3 and 2k8?

[winvarma]

Hi Durgesh,

we have this issue in 2k3 and 2k8 servers do you have any leads apart from enabling TLS at client end?

Regards

[durgeshsingh]

Hi

TLS need to enable both end client as well beacon. 

Here are some items to check on the 2003 server to help make sure it is using TLS 1.0

1. SCHANNEL for TLS 1.0 client and server is enabled and disabled by default: https://docs.microsoft.com/en-US/dotnet/framework/network-programming/tls#configuring-schannel-protocols-in-the-windows-registry

2. DefaultSecureProtocols: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp and/or HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp with a value of 0x00000080 for TLS 1.0

3. SchUseStrongCrypto: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319 and/or HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NetFramework\v4.0.30319 with a value of 1 to enable. This is only needed if only .NET 4.0 or higher is installed

4. SystemDefaultTlsVersions: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v2.0.50727 and/or HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v2.0.50727 with a value of 1 to enable. This is only needed if a version of .NET lower than 4.0 is also installed

5. TLS 1.0 via Internet Explorer: Settings->Advanced->Use TLS 1.0

6. Add beacon server to Local Intranet zone via Internet Explorer: Settings->Security

If you end up making any changes to the registry, you will also need to reboot the machine and then make sure that none of the changes are undone via group policy following the reboot. You can also use a tool like IIS Crypto (https://www.nartac.com/Products/IISCrypto) to compare the Protocols and Ciphers enabled on the beacon and the agent.

Another option would be to create a scheduled task to manually run ndtrack -t machine and then a follow up task to copy the NDI to the Incoming\Inventories folder on one of the beacons.

[craig_moore]

Just Windows 10 devices, as mentioned most work fine and others we get the error. 

We are testing against two Windows 10 laptops, one that works and one that does not - both laptops should have the same security settings as its set through group policy and confirmed that TLS 1.1 and 1.2 enabled.

[durgeshsingh]

Hi

Can you share installation & policy log from not working windows 10 machine.

Log can be find on below 

C:\Windows\temp\managesoft

[craig_moore]

Attached.

[durgeshsingh]

Hi 

I am unable to download these logs ,getting Failed - Forbidden error. 

can you make them .txt & then update.

[craig_moore]

[12/1/2021 11:22:19 AM (G, 0)] {5076} Download failure: An existing connection was forcibly closed by the remote host.
[12/1/2021 11:22:19 AM (N, 0)] {5076} Download FAILED for “https://beaconname/ManageSoftDL/Policies/Merged/otc.local_domain/Machine/taxl166hl63.npl?machinename=taxl166hl63&ipaddress=0.0.0.0,0.0.0.0,10.166.60.97,172.18.3.21,0.0.0.0”
[12/1/2021 11:22:19 AM (N, 0)] {5076} Downloading “https://beacon-name/ManageSoftDL/Policies/Merged/otc.local_domain/Machine/taxl166hl63.npl?machinename=taxl166hl63&ipaddress=0.0.0.0,0.0.0.0,10.166.60.97,172.18.3.21,0.0.0.0” to “C:\windows\TEMP\NDL29564.npl”
[12/1/2021 11:22:19 AM (G, 0)] {5076} Download failure: An existing connection was forcibly closed by the remote host.
[12/1/2021 11:22:19 AM (U, 0)] {5076} ERROR: Error (s107m858)
[12/1/2021 11:22:19 AM (U, 0)] {5076} ----------------
[12/1/2021 11:22:19 AM (U, 0)] {5076} The following network error occurred while retrieving the application:

An existing connection was forcibly closed by the remote host.

Contact your network administrator for assistance.

[durgeshsingh]

Thanks. Seems agent is not able to communicate its beacon. 

Can you run below command and share screenshot. 

C:\Program Files (x86)\ManageSoft\Policy Client\mgspolicy.exe -t machine -o UILevel=Auto

[ChrisG]

One possible cause of the error "An existing connection was forcibly closed by the remote host" is mismatch in TLS versions supported by the client and server, so doing troubleshooting around that sounds like a good path to consider.

Here are a couple of other troubleshooting steps to try to get more of a handle on what is and is not working (if you haven't already done this).

Try accessing one of the URLs the agent is using in a browser running on a problematic Windows 10 computer to see if the browser is able to access it. For example, try the URL https://your-beacon/ManageSoftDL/Policies/Merged/otc.local_domain/Machine/machine.npl?machinename=machine&ipaddress=10.166.60.97,172.18.3.21

Check whether a simple network connection to port 443 on the beacon can be made. For example, the following PowerShell command would test this:

Test-NetConnection your-beacon -Port 433

[craig_moore]

We were able to work on a couple of devices today and Test-NetConnection works but same An existing connection was 'forcibly closed by the remote host' error when we tried 

C:\Program Files (x86)\ManageSoft\Policy Client\mgspolicy.exe -t machine -o UILevel=Auto

As mentioned most devices report in fine but some do not and based on some initial testing we are thinking the difference is that device reports in fine when user is working remotely from home and same device fails when user is back working in the office on the corporate network.  We have just one beacon but heard that maybe security differences when in office versus working from home.

Still validating but if correct next step is to work with network/security team on a possible reason and options to resolve. 

Has anyone experienced something similar given the large number of working from home users during COVID?