Hi all,
I started monitoring IIS log at one of our beacons for HTTP errors and what is concerning me is that I'm seeing HTTP 404 error messages appearing every few minutes when FlexNet Agents are requesting "Default Machine Schedule.osd".
I tried switching in Beacon GUI back to local web server and then back to IIS but that didn't help. Anyone seen this and knows how to solve it?
Our installation is FNMS 2019R2 on-prem.
IIS Log:
2020-05-04 11:17:58 10.177.163.103 GET /ManageSoftDL/Schedules/Default+Machine+Schedule/Default+Machine+Schedule.osd - 80 - 10.185.145.216 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 404 0 0 140
BeaconIIS.log:
2020-05-04 06:17:58,582 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\Schedules\Default Machine Schedule\Default Machine Schedule.osd' not found.
Above record is from the same request, BeaconIIS log is in server time and IIS log is in UTC time zone.
Thanks!
Pavol
‎May 04, 2020 06:47 AM
Usually agents are getting the osd file from different folder with some generated GUID:
2020-05-04 11:37:22 10.177.163.103 GET /ManageSoftDL/Schedules/Default+Machine+Schedule+26E01C127ABB501658598FA2F3847162/Default+Machine+Schedule.osd - 80 - 10.179.162.34 Flexera+Inventory+Manager/13.1.1+(Linux) - 200 0 0 0
‎May 04, 2020 07:06 AM
Oh, now I found that it's not only about Default Machine Schedule.osd but also InventorySettings.ndc file:
2020-05-04 11:43:21 10.177.163.103 GET /ManageSoftDL/InventorySettingsConfiguration/InventorySettings/InventorySettings.ndc - 80 - 143.166.135.175 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 404 0 0 31
but in few minutes the same address was reachable with HTTP 200 OK:
2020-05-04 11:43:40 10.177.163.103 GET /ManageSoftDL/InventorySettingsConfiguration/InventorySettings/InventorySettings.ndc - 80 - 10.161.31.244 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 200 0 0 15
Maybe the Beacon is reaching it's limits?
‎May 04, 2020 07:09 AM
Below are logs from one Agent (10.177.163.103) requesting at three different times - first success, second error, third success. Why is that error happening?
Default+Failover+Settings.osd
2020-05-04 11:42:59 10.177.163.103 GET /ManageSoftDL/ClientSettings/Default+Failover+Settings+2B78CAEF2E02620554C3AF12C624CFF4/Default+Failover+Settings.osd - 80 - 10.160.235.30 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 200 0 0 15
2020-05-04 11:43:19 10.177.163.103 GET /ManageSoftDL/ClientSettings/Default+Failover+Settings+2B78CAEF2E02620554C3AF12C624CFF4/Default+Failover+Settings.osd - 80 - 10.160.235.30 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 404 0 0 15
2020-05-04 11:52:39 10.177.163.103 GET /ManageSoftDL/ClientSettings/Default+Failover+Settings+2B78CAEF2E02620554C3AF12C624CFF4/Default+Failover+Settings.osd - 80 - 10.160.235.30 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 200 0 0 0
InventorySettings.osd
2020-05-04 11:18:32 10.177.163.103 GET /ManageSoftDL/InventorySettingsConfiguration/InventorySettings/InventorySettings.osd - 80 - 10.160.235.30 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 200 0 0 15
2020-05-04 11:43:21 10.177.163.103 GET /ManageSoftDL/InventorySettingsConfiguration/InventorySettings/InventorySettings.osd - 80 - 10.160.235.30 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 404 0 0 62
2020-05-04 11:47:29 10.177.163.103 GET /ManageSoftDL/InventorySettingsConfiguration/InventorySettings/InventorySettings.osd - 80 - 10.160.235.30 FlexNet+Manager+Platform/13.1.1+(Windows+x86) - 200 0 0 46
‎May 04, 2020 07:19 AM
Oh, for the last comment I believe the errors were caused by Beacon generating new files. I just found this in BeaconIIS.log (time 6:43 is same as 11:43 in IIS logs)
2020-05-04 06:43:08,765 [ment.PolicyFileWatcher] [INFO ] Policy file watcher detected a new beacon policy
2020-05-04 06:43:08,765 [mmon.BeaconPolicyCache] [INFO ] Loading beacon policy from 'C:\ProgramData\Flexera Software\Beacon\BeaconPolicy.xml'
2020-05-04 06:43:08,765 [mmon.BeaconPolicyCache] [INFO ] Certificate configurations not found.
2020-05-04 06:43:08,765 [mmon.BeaconPolicyCache] [WARN ] Failed to execute the method: <LoadBeaconPolicy>b__0 with error:
The process cannot access the file 'C:\ProgramData\Flexera Software\Beacon\BeaconPolicy.xml' because it is being used by another process.). Attempt 1 of 10 retries.
2020-05-04 06:43:09,312 [mmon.BeaconPolicyCache] [INFO ] Loading beacon policy from 'C:\ProgramData\Flexera Software\Beacon\BeaconPolicy.xml'
2020-05-04 06:43:09,312 [mmon.BeaconPolicyCache] [INFO ] Certificate configurations not found.
2020-05-04 06:43:09,343 [ment.PolicyFileWatcher] [INFO ] Clearing beacon web server cache, because a change to the beacon policy file was detected.
2020-05-04 06:43:09,343 [eployment.CacheManager] [INFO ] Clearing cached items from disk
2020-05-04 06:43:18,843 [eployment.CacheManager] [WARN ] Failed to clean up file cache path 'C:\ProgramData\Flexera Software\Staging\Common\Policies'.
System.UnauthorizedAccessException: Access to the path 'ausdwanlapp02-50230DF7ECDAB19ECD6CA367F8EED6BCA777C353.npl' is denied.
at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
at Flexera.Beacon.Deployment.CacheManager.ClearFileCache()
2020-05-04 06:43:18,874 [ation.PackageGenerator] [INFO ] Generating package of type 'ClientSettings' at 'ClientSettings'
2020-05-04 06:43:18,890 [ntFormats.AgentPackage] [INFO ] Writing package file, folder='C:\ProgramData\Flexera Software\Staging\Common\ClientSettings\Default Failover Settings', pkgName='Default Failover Settings', extension='ndc', internalPackageName='ManagedDeviceSettings'
2020-05-04 06:43:19,156 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\ClientSettings\Default Failover Settings 2B78CAEF2E02620554C3AF12C624CFF4\Default Failover Settings.osd' not found.
2020-05-04 06:43:20,999 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\ClientSettings\Default Failover Settings 2B78CAEF2E02620554C3AF12C624CFF4\Default Failover Settings.osd' not found.
2020-05-04 06:43:21,718 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\InventorySettingsConfiguration\InventorySettings\InventorySettings.osd' not found.
2020-05-04 06:43:21,781 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\InventorySettingsConfiguration\InventorySettings\InventorySettings.ndc' not found.
2020-05-04 06:43:22,015 [ation.PackageGenerator] [INFO ] Generating package of type 'InventorySettings' at 'InventorySettingsConfiguration'
2020-05-04 06:43:22,015 [ntFormats.AgentPackage] [INFO ] Signer is null. Signature verifcation is skipped
2020-05-04 06:43:22,031 [ntFormats.AgentPackage] [INFO ] Writing package file, folder='C:\ProgramData\Flexera Software\Staging\Common\InventorySettingsConfiguration\InventorySettings', pkgName='InventorySettings', extension='ndc', internalPackageName='InventorySettings'
2020-05-04 06:43:22,046 [ation.PackageGenerator] [INFO ] Generating package of type 'ClientSettings' at 'ClientSettings'
2020-05-04 06:43:22,046 [ntFormats.AgentPackage] [INFO ] Writing package file, folder='C:\ProgramData\Flexera Software\Staging\Common\ClientSettings\Default Failover Settings 2B78CAEF2E02620554C3AF12C624CFF4', pkgName='Default Failover Settings', extension='ndc', internalPackageName='ManagedDeviceSettings'
2020-05-04 06:43:22,062 [ation.PackageGenerator] [INFO ] Generating package of type 'ClientSettings' at 'ClientConfiguration'
2020-05-04 06:43:22,062 [ntFormats.AgentPackage] [INFO ] Writing package file, folder='C:\ProgramData\Flexera Software\Staging\Common\ClientConfiguration\Device Configuration 9BC0313A42B83B0E845936DBF790BAC9', pkgName='Device Configuration', extension='ndc', internalPackageName='Device Configuration'
2020-05-04 06:43:22,062 [ation.PackageGenerator] [INFO ] Generating package of type 'Schedule' at 'Schedules'
2020-05-04 06:43:22,062 [ntFormats.AgentPackage] [INFO ] Writing package file, folder='C:\ProgramData\Flexera Software\Staging\Common\Schedules\Default Machine Schedule 26E01C127ABB501658598FA2F3847162', pkgName='Default Machine Schedule', extension='nds', internalPackageName='Default Machine Schedule'
2020-05-04 07:02:20,142 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\Schedules\Default Machine Schedule\Default Machine Schedule.osd' not found.
2020-05-04 07:12:34,714 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\Schedules\Default Machine Schedule\Default Machine Schedule.osd' not found.
2020-05-04 07:12:37,276 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\Schedules\Default Machine Schedule\Default Machine Schedule.osd' not found.
2020-05-04 07:15:54,238 [s.DownloadDLController] [WARN ] Requested file 'C:\ProgramData\Flexera Software\Staging\Common\Schedules\Default Machine Schedule\Default Machine Schedule.osd' not found.
‎May 04, 2020 07:26 AM
I starting to get a feeling that monitoring IIS log for 404 errors is useless...
‎May 04, 2020 07:28 AM
Hi Pavol,
When agents request policy, they get a unique personalized policy. The agent policy file contains paths to various other packages (OSDs). The agent packages are created for a unique set of properties. Agents with same set of properties get the same package. E.G. agents part of same Inventory Groups get the same Default Failover Package, agents with same schedule(s) get the same Default Machine Schedule package and so on. The Inventory Settings package is same for all agents.
When an agent requests a policy, Beacon evaluates its various set of properties and for each package, it checks if the files already exists, otherwise it creates the file, and then serves the policy. When a Beacon downloads a new Beacon Policy file, it can affect agent's properties, hence Beacon deletes all the local packages and they are rebuilt when agents request their policies.
If the Beacon Policy file is updated between an agent requests the policy and Beacon updates its own policy file, then the agents may get a 404 error. In that case they try with other Beacons, while it is not guaranteed that other Beacons would have the package either. So the IIS errors you see may because the Beacon ran into that situation OR because another Beacon ran into that situation and agent is requesting this Beacon as a failover request.
To resolve, stop the Beacon webserver and delete the following folders at C:\ProgramData\Flexera Software\Staging\Common
And then restart the BeaconIIS. If possible, have an agent request the policy or just monitor the logs for some time:
When agent requests the policy, the BeaconIIS log shall show messages like "Writing package file" and you should not see an issue with packages it is creating. However if an agent requests a package, without actually requesting policy first, from this Beacon it may still throw a 404.
Hope that helps.
‎Jul 02, 2020 01:25 AM