Reconcile takes 7 to 13 hours to complete

[IronManMK10]

Good afternoon

I have a reconcile in FNMS takes around 7 to 13 hours to complete.

Is there any thing that we can check to improve reconcile time. The DB is optimised to the best that we have.

The below step takes over 5 hours to run.

2022-12-07 06:38:31,941 [INFO ] WriteFileEvidence - Match double ended wildcards
2022-12-07 06:38:31,941 [INFO ] Writer step 'WriteFileEvidence - Match double ended wildcards' with non-default execution path (Fallback) meets condition TestingOrFallback.
2022-12-07 12:29:55,700 [INFO ] Matched 1554186 records to existing File Evidence with start and end wildcards
2022-12-07 12:29:55,700 [INFO ] Successfully processed in 5 hours, 51 minutes, 21 seconds

I have attached the log accordingly.

[mfeinman]

Thanks for the attention this issue is getting.

In my case, the issue started with the 12/4 reconciliation.  My ARL was updated on 12/3 (and is done weekly on Saturdays).  (I originally thought it had something to do with my agent rollout but I now see that was just a big coincidence.)

Edit: I am using FNMS 2019 R1 on prem (server upgrade is planned in a few weeks).

--Mark

[mfeinman]

Hi all ... My reconciliation process that ran last night (Sunday 12/11) was back to normal.  Total time was about 6 1/4 hours and the double-wildcard step ran in 54 minutes:

2022-12-12 04:57:36,288 [INFO ] WriteFileEvidence - Match double ended wildcards
2022-12-12 04:57:36,288 [INFO ] Writer step 'WriteFileEvidence - Match double ended wildcards' with non-default execution path (Fallback) meets condition TestingOrFallback.
2022-12-12 05:52:21,608 [INFO ] Matched 40923 records to existing File Evidence with start and end wildcards
2022-12-12 05:52:21,608 [INFO ] Successfully processed in 54 minutes, 45 seconds

(Is 54 minutes a decent time to match on 40,923 records?)

--Mark

[ChrisG]

Thanks for confirming the import time is back to normal - that's great to know!

The time taken by "WriteFileEvidence - Match double ended wildcards" step largely depends on how many unique file paths & names have been gathered from your managed devices, which in turn is typically broadly proportional to the number of devices you are managing. The time taken is not proportional to the number of "matched records" that the step finds.

Based on your observation of this step taking 54 minutes, I would guess that you have file evidence from some 10s of thousands of devices.

[mfeinman]

@ChrisG , your guess was good.  About 38,000 devices.

--Mark

[nrousseau1]

OK, thanks @IronManMK10  and @diaz06  for sharing. The issue is not isolated, happens after the last ARL update. We are going to consider removing these Tibco evidences from tomorrow's ARL before looking for better solutions.

The "ignore evidences" approach I provided earlier will not enhance performance as the matching step will still happen. It does not hurt to try but removal of the evidences will probably be the only solution.

[IronManMK10]

Hi @nrousseau1 

We have already marked them as ignored. Can we delete them from the Ignored Evidences page? or should we wait for the new ARL.

[nrousseau1]

ARL file evidences are referenced in Multiple places (SoftwareTitleFileEvidence, ImportedFileEvidenceMapping), so, a simple DELETE FROM FileEvidence (Or NewFileEvidence_MT) would not work.

Tomorrow's ARL will have the Tibco File evidence removed, this is confirmed.

I would rather update the ARL than make a dangerous SQL Deletion.

[IronManMK10 · Accepted solution]

After the ARL release, the issue has been resolved. Many thanks for assistance and prompt resolution.