cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Seeking Assistance @ VCenter Introspection Rules and IBM PVU Scanning of VMWare Vcenter

Hello Everyone, I have been trying to understand why some of our VCenter Introspection Rules under the Discovery and Inventory Section never show any  numbers in the "Service Discovered", "Inventory Completed", "Inventory Skipped" or "Inventory Failed"  even though when I look at the logs available in the FNMS tool and the logs available on the Beacons ... devices are being discovered and inventoried by these Rules.

I have been analyzing data between what rules work and what rules don't and have not come up with any answers as of yet. The "action" in the rule is all the same of "Vcenter Introspection" for each rule so that is not it. Targets are targets so that should not be the difference.

What I have noticed is that the times in the Next and Last Run seem to be a little wonky.

It looks like the IBM PVU Scanning of the VMWare VCenters (every 30 minutes), is combined with the Vcenter Rules running (which we have scheduled for once a day generally at night time) which maybe skewing things .. don't know. 

The other point I thought of, which I do not know the answer to is ..  what happens if either the "Rule" or "IBM PVU VCenter" scans do not find any changes ..  perhaps in my scenario the 30 minute IBM PVU VCenter scans are zeroing out any previous numbers that our nightly Rules are finding.

I apologize in advance for the length of this .. but its driving me crazy on why I see this behaviour.

 

Thx as Always

Bruce

 

(10) Replies

Hello Again, just wanted to add that we have 5 Beacons which participate in these VCenter rules depending on the subnet of the Vcenter. The only Rule that indicates numbers in the FNMS tool is our AsiaPac one which is 12 -13 hours ahead of us here in Eastern time. Analyzing some more logs, is leading me to think that because we have 5 potential Beacons (depending on the subnet) which attempt every rule, the Beacons that do not participate in the VCenter scanning, report as completed and perhaps update the FNMS tool with zeroes.

Once again, a little bit of knowledge is very dangerous and I can only find a little bit of detail in the System Reference Guide pertaining to how these VCenter scans actually operate in conjunction with the IBM PVU VCenter scanning .. if anyone has come across any literature that explains how these scans operate, I would really appreciate it.

 

Hi @bruce_giles 

To check the vCenter inventory, one point to start should be to check the logs on beacons.

Check c:\programdata\Flexera Software\compliance\logging\inventory rule

Here you have all the logs run by the beacon related to inventory, order by date, and open the most actual log.

in VmWareInventory.log you will find what is trying to do the beacon.

You can search and check if you find your vCenter here and see what errors are returning.

Some consideration to take is the if ICMP (ping) protocol is disabled the automatic discovery of the vCenter will not work and will fail, you need to modify the inventory rules in UI and disable the automatic discovery.

If port 443 is closed again the inventory will fail and will say that discovery did not find any service on the device.

Regarding PVU scanning, I can't advise here as I didn't activate this in my environment.

 

Hi Bruce,

So if I understand correctly, only 1 vCenter is currently working right?

The IBM PVU scanning enables the frequent vCenter scanning on all vCenter targets (in 2019 R2 you can select your targets from a dropdown list). in FNMS versions before 2019 R2 you can select the option "Perform remote scans against all known VMware vCenter and Oracle VM Manager servers" in the inventory settings, this will create a rule on all the beacons for scanning the vCenter every 30 minutes. Note, the IBM PVU scan depends on the configuration of the normal vCenter scan in FNMS. (targets, subnet assignments, passwords firewalls etc)

I would recommend the following

- Check if the normal vCenter scan is working on the beacon

-  Check your targets and subnet assignments (I would always create dedicated /32 subnets for scanning vCenter which are assigned to a dedicated beacon)

- verify the inventory rule log (discovery.log contains the name of the task, this should match the IBM PVU rule)

Hope this helps

Stefan

 

Hi Guys .. thanks for the suggestions .. here is the thing though . When I look at all of the logs on the Beacons, both the IBM PVU VCenter and the VCenter Introspection Rule processes work as expected on all rules and indeed find the vcenters and perform the inventory scans. I have 2 rules that indicate that services have been discovered and inventoried in the FNMS GUI under the Discovery and Inventory rules but the others do not.

Here is an example of what I am seeing ... I have attached a Word Document with the Image from one of the rules. So, the Rule ran this morning at 2:08 am and found the VCenters and identified them in the tool. Later today, about 4 or 5 hours from now, the screenshot I attached will be updated with all zeroes and a time that indicates that same 4 or 5 hours.

So, I am thinking that the IBM PVU VCenter scan that runs every 30 minutes is clobbering these numbers. This is where I do not know what exactly happens during these scans if nothing has changed. Perhaps it updates things .. not sure.

In Summary then .. everything is working for all rules according to the logs but do not indicate that in FNMS tool except for 2 rules .. one that runs in AsiaPac which maybe the 13 hour time difference is affecting and the one that I have attached in the Word document which the data changes 4 or 5 hours later.

No doubt its probably the way we have things setup or the fact that our Vcenter team reassigns Hosts and VM's on a nightly basis along with the fact that we have 5 Beacons and all 5 are running these VCenter scans every 30 minutes along with our nightly rule. Just trying to understand why we are seeing what we are. I think I am onto something with the example I attached since the numbers are there for a bit but then get updated to all zeroes when what looks to be the IBM PVU scans are running. Not sure if the IBM PVU scans update the Inventory Rules or not.

 

Again, sorry for all the details .... 

 

Bruce

 

 

Forgot the Attachment on last post ...

Hello There, as predicted ..  the Rule has been updated at 10:06 am  and has zeroed out the numbers previously recorded. I will look into the logs to see if I see something. Will probably submit a Case for this and have someone look at our setup .. for sure something is going on.

 

I have attached the Word document again .. with both screen shots

 

Will update the post when I receive some more details ...

 

Bruce

I know the advice previously was to check the VMWareInventory.log but have you also checked the Discovery.log as well?

If your rule was a Discovery and Inventory rule then it must complete discovery first before inventory collection on vCenters will commence, this means that if there's an issue with Discovery then it would cause the task not to have inventory.

IBM PVU currently pre-populates what vCenters to scan so doesn't have quite the same issue however it is also more inclusive than it needs to be, we have a couple of improvements coming in the next couple of cloud releases that will restrict the discovery our PVU rule carries out to just vCenter but also to give more control over what targets the 30 minute vCenter scan applies to.

 

So if the discovery log for your main scan shows a small number of devices, then these upcoming changes may help.

If you can't see anything in either the discovery or VMInventory logs then please do raise a case.

(Anything expressed here is my own view and not necessarily that of my employer, Flexera)
If the solution provided has helped, please mark it as such as this helps everyone to know what works.

Hi mrichardson,  thanks for the reply and providing some insight into what changes are forth coming, really appreciate it.

After digging around some more I think I have solved the mystery for my specific situation. It is a little complicated but I will try and simplify it the best I can.

My concern was  related to the fact that when I looked at the Summary Table within the VCenter Rules that I created under the Discovery and Inventory Rules .. the tables with the exception of two would always reflect zeroes when I looked at the start of my day at 7 am Eastern Time. As identified earlier in this thread I found that after 10 am Eastern time, one of my rules would get zeroed out after earlier having numbers populated.

Long story short, we have 5 Beacons which reside across the Globe .. our AsiaPac Beacon due to the 16 hour time difference versus me here in Eastern Time Canada, is the Beacon which is updating these tables to zeroes with the exception of the rule that is specifically for AsiaPac. I did not realize that all Beacons attempt to run all of the rules when they are scheduled. Of course, not all will "actually" perform the discoveries and inventories because of the subnet assignments. What I have concluded is that even though the Beacon does not perform the discovery  and inventory it does indicate "completed" and as such updates the table for that Rule.

In my scenario, when the AsiaPac Beacon runs all of my Rules and the fact that it is 16 hours ahead of me (like next day in some cases) it updates the tables to zero before I get into the office and logged onto the FNMS Tool. I had the one rule which I would see change because the starting time of that rule fell within my working hours. The other thing I will mention is that my rules are scattered in their starting times which added to the chaos when I was trying to analyze things.

All of the logs, both discovery and inventory do indicate that the devices are being discovered and inventories run, but I didn't see those reflected in the FNMS tool because of what I have tried to explain above.

 

Thx again for all of the input .... I knew there had to be an explaination.

 

Bruce

Hi Bruce,

What you describe is a known issue, as you say the beacons that run in local time then report up to the app server and are resolved into the DB which is a different time zone and interprets the time in the activitystatus files using it’s own time which could be earlier or later than the time the beacon says it started.

I’ll check in the office tomorrow what the status of that bug is.

As a FYI if you see beacon tasks remaining in “In Progress” indefinitely then this is likely the cause of that also.

The only current workaround is to set the time settings of all beacons to the same timezone as the DB server.
(Anything expressed here is my own view and not necessarily that of my employer, Flexera)
If the solution provided has helped, please mark it as such as this helps everyone to know what works.

Hi mrichardson, thanks once again for the confirmation .. really appreciate it. It is later on in the evening for me so I logged onto my FNMS Tool from home to check my suspicions and sure enough, for the rules that have run during the past hour or two ..  the appropriate numbers are indeed identified in the table on the Discovery and Rules area as expected. So, as you confirmed in your last post, its our AsiaPac Beacon which runs 16 hours ahead of our other Beacons which updates the tables to zeroes. As you suggested, I could re schedule things if need be .. but my main goal was to simply understand why I was seeing what I was.

It's been a good learning exercise for me and hopefully for some others who are in a similar position,  where we know enough to be dangerous.

Thx to Everyone, I kind of thought it was probably something on our side which was influencing things ..

Bruce