FNMS Cloud Reconciliation taking longer and longer each day.

[EHacking]

This is for FlexNet Manager in the cloud. Is anyone else seeing delayed reconciliations?

Yesterday's reconciliation didn't even kick off until 4:00 p.m., and today's reconciliation hasn't started either.

It appears the holdup is the FlexNet Manager Suite Inventory Import. Yesterday ours kicked off at 3:17 a.m. and finished at 5:20 p.m. This Import kicks off the daily Reconciliation. 

What is being done by Flexera to resolve this issue? When looking at the past week or so our Inventory Import and Reconciliation is taking anywhere from 10 to 13 hours, and usually ends after our work day has ended.

[mrichardson]

Hi @EHacking 

I believe you've already got a case open for your specific environment, I would recommend the first place to check if you believe there are ongoing issues impacting all customers is https://status.flexera.com/

If you haven't already, you can subscribe to receive updates when any major issues are reported on the cloud, this site is updated when we confirm an issue significantly impacts multiple customers.

[ChrisG]

Looking at the timing of this post, it sounds likely that the observations here may at least partly have been related to the following incident: FlexNet Manager Suite Cloud - some customer Licensing positions may be unavailable or out of date

[EHacking]

Looks like the issue may be resolved after last night's 2019 R2.1 upgrade. We will monitor for the next couple of days. I was just curious if others were experiencing this.

[KPBussey]

Good to hear it's looking better, @EHacking. And, thank you for posting - it's helpful for others to see and, if someone runs into something like this down the road, they know someone else has experienced it. (Note - hoping no one else does experience it!)

[bmaudlin]

@EHacking We were experiencing the same issue also, and was fixed by the hot fix that was applied, and performance is returning to normal.

@KPBussey I have discussed this before with @mrichardson when there is an issue with the Flexera cloud which impacts multiple customers. I think it's worthwhile making the status page more visible than it is currently, as I don't think there is a link to it from the Community page. If there is a severe outage, then a banner or such like on the Community page I feel would be a good idea.

Ben 

[bmaudlin]

Incidentally - I am registered for updates on the Status page, and I received no update to state there was an issue or that it was fixed.

[dosborn]

Hi @bmaudlin , we have recently added an additional component, "Batch processing", to our Status Page. I believe you may need to add it to your existing subscriptions before you will be advised of any incidents impacting that service.

[bmaudlin]

@KPBussey that's great to know that is being looked into.

@dosborn thank you for the heads up, I've added that field to my status updates.

[KPBussey]

@bmaudlin thanks for the feedback - it's good timing. @dosborn  had reached out to me to discuss how we can improve visibility of the System Status Dashboard within the community. For time being, there is a link to it in the right-rail of the page. We are aiming to have something with more visibility in early 2020. 

[EHacking]

I think the issue is not really the actual reconciliation, which appears to take only 15-30 minutes now. It is the time between "Import into staging" and "Import Inventory devices." This can take anywhere between 4 hours and 15 hours. See attached.

Longest processing time in the past week:

Step	Start	Finish	Total	Time between
FlexNet Manager Suite	12/4/2019 9:37 AM	12/5/2019 3:52 AM	18:15
Import into staging	12/4/2019 9:37 AM	12/4/2019 10:28 AM	0:50
Import inventory devices	12/5/2019 2:27 AM	12/5/2019 3:36 AM	1:09	15:58
Reconcile  software licenses	12/5/2019 3:36 AM	12/5/2019 3:52 AM	0:16

Today's time

Step	Start	Finish	Total	Time between
FlexNet Manager Suite	12/11/2019 2:50 AM	12/11/2019 9:40 AM	6:50
Import into staging	12/11/2019 2:50 AM	12/11/2019 3:21 AM	0:30
Import inventory devices	12/11/2019 8:42 AM	12/11/2019 9:18 AM	0:36	5:21
Reconcile  software licenses	12/11/2019 9:18 AM	12/11/2019 9:40 AM	0:21

[ChrisG]

The time gaps between steps you have observed are likely reflective of the time it took from the relevant job being submitted at the end of the previous step to the job reaching the front of the queue to be processed (the "queue wait time").

The Flexera team is now monitoring that metric as a service level indicator. Here is a snapshot of recent queue wait times to give some transparency around how this looks:

(The numbers on the left axis of the "Current queue wait time" chart are minutes.)

[EHacking]

Thanks @ChrisG for the info. But this still doesn't get our reconciliation completed before start of work day. I'm hoping things will level off over the holidays. Maybe in the New Year we can see our true license position by at 7 or 8 a.m.

[mrichardson]

Hi @EHacking,

There is work going on to clean up some of the old usage data in the Inventory database which likely is having an impact on the steps you mention.

Once this is completed you should see a reduction in those times so please bear with us and keep an eye on it over the next week or so.

[EHacking]

Spoiler

@mrichardson Thanks! Today's total time was only 2:14, so the time is definitely dropping. I will continue to monitor.

[auntalan0]

@ChrisG , That chart you posted here with the SLA metrics, is there a place or URL I can go to to see the Avg queue times so i can get an idea of when my jobs may kick off relative to when I pushed the button or the schedule fired off?

[ChrisG]

@auntalan0 - The particular chart I posted is from the systems that the Flexera Site Reliability Engineering team use internally at Flexera for monitoring the FlexNet Manager Suite SaaS environment. However you can see times that different events kicked off and completed for your own tenant by using the System Tasks page when you are logged in to FlexNet Manager Suite.