Inventories uploaded by full Kubernetes agent

[aaaaaa]

Hello,

I have install full K8S agent in cluster. May I know which components (controller or nodes) will generate and upload *.ndi files of K8S inventory and how many .ndi files will be generate? Is it have command to run manually to check if it upload success or not?

Do they be uploaded on beacon the same time with other agents?

Thank you.

[Colvin · Accepted solution]

The "monitor" component is responsible for all interactions with the beacon, including the upload of .ndi files. It will generate .ndi files for each node in the cluster (a standard hardware-only inventory), for each discrete container image it observes, and one for the observed pods, nodes (containing Kubernetes metadata, as opposed to the hardware inventories), and namespaces. It will upload these .ndi files once 15 minutes after startup and then ongoing on a 24 hour interval. It does not currently consume the standard agent schedule, but can be configured for different upload intervals. The FNMS documentation covers the spec.monitor.interval attribute of the agent config and how that would be set.

You can view the Kubernetes agent logs using the kubectl logs command as you would to view any other container's logs. It will emit log messages each time it attempts an upload and will report an error if it fails. You can search the logs for the term "upload" or "uploader". It may be necessary to view the uploader.log file that is generated by the ndupload component of the agent. This is stored in the same location as in a standard deployment, so you can either use kubectl cp to copy it out of the agent's monitor container or you can use kubectl exec to run the cat command to view its contents.

kubectl logs -n flexera krm-instance-monitor-0 | grep uploader
kubectl cp -n flexera krm-instance-monitor-0:/var/opt/managesoft/log/uploader.log uploader.log
kubectl exec -n flexera krm-instance-monitor-0 -- cat /var/opt/managesoft/log/uploader.log

[Colvin · Accepted solution]

The "monitor" component is responsible for all interactions with the beacon, including the upload of .ndi files. It will generate .ndi files for each node in the cluster (a standard hardware-only inventory), for each discrete container image it observes, and one for the observed pods, nodes (containing Kubernetes metadata, as opposed to the hardware inventories), and namespaces. It will upload these .ndi files once 15 minutes after startup and then ongoing on a 24 hour interval. It does not currently consume the standard agent schedule, but can be configured for different upload intervals. The FNMS documentation covers the spec.monitor.interval attribute of the agent config and how that would be set.

You can view the Kubernetes agent logs using the kubectl logs command as you would to view any other container's logs. It will emit log messages each time it attempts an upload and will report an error if it fails. You can search the logs for the term "upload" or "uploader". It may be necessary to view the uploader.log file that is generated by the ndupload component of the agent. This is stored in the same location as in a standard deployment, so you can either use kubectl cp to copy it out of the agent's monitor container or you can use kubectl exec to run the cat command to view its contents.

kubectl logs -n flexera krm-instance-monitor-0 | grep uploader
kubectl cp -n flexera krm-instance-monitor-0:/var/opt/managesoft/log/uploader.log uploader.log
kubectl exec -n flexera krm-instance-monitor-0 -- cat /var/opt/managesoft/log/uploader.log

[aaaaaa]

Thank you for your reply @Colvin ,

I've checked it uploaded successfully. But inventories are not imported but appeared in badlogs folder because of "unknown" value in tracker field.

I found an 'known issue' article and it has a hotflix to solve this problem but I'm not sure it support on FlexNet Manager Suite release: 2022 R1 (On Premises) or not.

Do you know about this?

Thank you.

[ChrisG]

Here are details of the known issue that I believe you are referring to here: NDI files from the Kubernetes inventory agent may fail to import due to the word "unknown" appearing as the agent version (IOJ-2215285)

As per the information on that page, there is a hotfix for the Kubernetes agent available for download from the Product & License Center. Look in the "FlexNet Manager Suite 2021 R1" download package folder for the hotfix for that release.

[aaaaaa]

Hello @ChrisG ,

I just install a hotfix to fix the error K8S agent inventories cannot import because of "unknown" value in Tracker field but it not work. Inventories still go to Badlogs folder.
I've followed intructions in README file of hotfix package: run deploy.ps1 on web application server.

Do I need to install another version of k8s agent? (we're using version 1.0.1, but I see in hotfix package is 1.0.3)

Please help me fix this.
Thank you.

[ChrisG]

The agent version 1.0.1 is the version that contains the "unknown" issue. The hotfix version 1.0.3, so that is what you will need to use.

If you are still seeing "unknown" appearing in the Tracker field in NDI files then I think that indicates you are still have the 1.0.1 agent installed on the Kubernetes infrastructure.

[Colvin · Accepted solution]

The process for deploying an updated version of the Kubernetes agent is nearly identical to the process for installing it in the first place. You will 1) import the new container image into the registry (the image tag is the version number) and 2) run the install.sh script (using the same command-line flags as before). You do not need to run the generate.sh script or apply the resulting krm.yaml file to the cluster.

Technically, by running the install.sh script you are modifying the controller component's image so it uses the new version. It will be restarted using the new image and then detect a variance between its image version and that used by the other agent pods. It will then update the other pods to use the new image version and they will restart. Once everything has restarted it will be running the new version.

[guitms]

If the upload is correct, according to the logs. Where will they end up in the Application server, because I don't see it in the All Containers section? We're running v1.0.3.

[aaaaaa]

They'll appear both  in All containers and All inventories. If you don't see any results, you should check the inventories folder on Application server and Web resolvers log, may be they got errors and went to Badlogs folder.

[guitms]

Thanks aaaa,

According to the install log on the K8s cluster last monday, it looks ok and it seems that the ndi's are send to the beacon. But if I check on our App server,  nothing found for Id or Clustername. I've searched through the whole dir structure and these names are not even popping up. Weird. Badlogs folder is empty, so the format of the ndi's should be ok. Any ideas?

$ k logs pod/krm-flexera-node-v5mss
krm 2022-04-11T13:33:21.117851079Z INFO -- version 1.0.3
krm 2022-04-11T13:33:21.118155189Z INFO -- begin
krm 2022-04-11T13:33:21.118173434Z INFO -- running on node df8145ad-051f-4bc7-8936-3345e8d76114
krm 2022-04-11T13:33:21.118178974Z INFO -- upload uri http://flexera.flexera.svc/ManageSoftRL
krm 2022-04-11T13:33:21.133505549Z INFO -- uid f61ba962-e2b0-42e4-84f7-752ea1b1af3e
krm 2022-04-11T13:33:21.133528117Z INFO -- resource version 136276066
krm 2022-04-11T13:33:21.133536912Z INFO -- testing the upload endpoint
krm 2022-04-11T13:33:22.158654671Z INFO -- upload endpoint looks ready
krm 2022-04-11T13:33:22.158970218Z INFO -- running inventory
krm 2022-04-11T13:33:28.416944584Z INFO -- inventory completed and uploaded
krm 2022-04-11T13:33:28.4170911Z INFO -- starting loop with interval 30m0s

[aaaaaa]

Hi guitms,

How long did you set for inventories upload? You should check uploader.log on krm-monitor pod to see if it has any error.

You could also check the upload logs on beacon too.

If everything is okay, let try reconcile again to check the database import compliance process.

Regards,

[guitms]

THanks A. We left everything default during installation of the krm controller. WIll check the uploader.log once the ClusterAdmins have setup my role correctly. 😉

Thanks for your help.

[guitms]

Now I know why I don't see anything in my FNMS Gui. I see this in the uploader.log:

[Wed Apr 13 13:33:21 2022 (N, 2)] {117} Prioritizing servers for upload
[Wed Apr 13 13:33:21 2022 (N, 2)] {117} Starting to prioritize servers from "HKLM\Software\ManageSoft Corp\ManageSoft\Common\UploadSettings"
[Wed Apr 13 13:33:21 2022 (N, 2)] {117} Applying prioritization algorithm "MgsRandom" with parameters ""
[Wed Apr 13 13:33:21 2022 (N, 2)] {117} Applying prioritization algorithm "MgsPing" with parameters ""
[Wed Apr 13 13:33:21 2022 (N, 2)] {117} Applying prioritization algorithm "MgsADSiteMatch" with parameters ""
[Wed Apr 13 13:33:21 2022 (N, 2)] {117} SelectorAlgorithm "MgsADSiteMatch" could not be loaded
[Wed Apr 13 13:33:21 2022 (N, 2)] {117} ERROR: Could not successfully prioritize servers
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Upload directory is '/var/opt/managesoft/tracker/inventories'
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Will attempt to upload file(s) 'k8s-node-ed5d3ef0-f61ba962-20220411T133322.ndi'
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Connecting to upload location 'http://flx-beacon.intranet.rws.nl/ManageSoftRL/Inventories'
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Upload username is ''
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Upload proxy is ''
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Uploading file 'k8s-node-ed5d3ef0-f61ba962-20220411T133322.ndi' to 'http://flx-beacon.intranet.rws.nl/ManageSoftRL/Inventories'
[Wed Apr 13 13:33:21 2022 (N, 0)] {117} Error 0xE000006F: Connection refused
[Wed Apr 13 13:33:21 2022 (N, 0)] {117} Error 0xE050044D: Failed to create remote directory /ManageSoftRL
[Wed Apr 13 13:33:21 2022 (N, 0)] {117} Error 0xE0690099: Specified remote directory is invalid, or could not be created
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} ERROR: Remote directory is invalid
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Uploading finished
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} ************************************************************
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} Program exited with code 1
[Wed Apr 13 13:33:21 2022 (G, 0)] {117} ************************************************************
root@krm-flexera-monitor-0:/var/opt/managesoft/log#

Looks like something on the App server is not correct? Am I correct?

[ChrisG]

A "Connection refused" error like you're seeing here would often indicate that something in the networking setting up not allowing a network connection from the device/node to the beacon.

To test that, verify that the node is able to make a network connection to port 80 (HTTP) on flx-beacon.intranet.rws.nl.