Re: Import and Reconcile takes lot of time after upgrade Flexnet Manager Suite to 2019 R1

SandeepKolhe · ‎Dec 05, 2019

We have upgraded Flexnet Manager Suite on-premise to 2019 R1 (from 2016 R1), post upgrade import/reconcile taking around 12-13 hours, before upgrade it was completing in 5-6 hours. Due to this long import timing, users are getting performance issue to access FNMS UI. We were executed index re-build script provided in Flexera upgrade documents as well.

Can someone please suggest how we can fix such issues.

Best regards,

Sandeep

ChrisG · ‎Dec 06, 2019

Some increase in import time between the 2016 R1 and 2019 R1 releases would not be surprising: the 2019 R1 release contains a lot more functionality, which - assuming no change in hardware - will place more load. However the increase you have described sounds larger than I would expect.

You may get some further insight into what is going on by taking a look at the import logs from before and after the upgrade to identify which import steps are taking the longest time now and comparing that to which steps were taking the longest time before the upgrade.

(Did my reply solve the question? Click "ACCEPT AS SOLUTION" to help others find answers faster. Liked something? Click "KUDO". Anything expressed here is my own view and not necessarily that of my employer, Flexera.)

mrichardson · ‎Dec 09, 2019

Further to what Chris mentions, there are several steps which will take longer in the short term, for example we improved the identification of orphaned file evidence and ensured it is removed; this was across both 2018 R2 and 2019 R1.

When you first upgrade, the steps to remove orphaned file evidence in particular will take a while if you have large amounts to delete but once done, it should be quicker going forward.

Therefore if you can identify the top 3-5 steps which are taking a long time we can better identify what the root cause might be and whether it's something that will only take longer in the short term.

(Anything expressed here is my own view and not necessarily that of my employer, Flexera)
If the solution provided has helped, please mark it as such as this helps everyone to know what works.

SandeepKolhe · ‎Dec 09, 2019

Thanks all for your quick responses for this issue.

We have gathered DB statistics and rebuild indexes, now we are monitoring all import processes and comparing it with before upgrade timings. I will share details about import timing in couple of days.

Thanks, Sandeep

+1 (612) 442-4276

mfranz · ‎Dec 12, 2019

As @ChrisG mentioned, you should really check the logs. For example: Do you import from SCCM? There used to be an issue with the number of Access Evidences from SCCM growing with each import.

mrichardson · ‎Dec 12, 2019

Agree totally with @mfranz on this, typically what support do in these scenarios is first look in the logs and check for:

What step is taking a long time (look at the timestamps to see long gaps)
Which data source is being imported from during that step.

Once we know these 2 things, we can check what information the reader is pulling in and what it's doing to identify if there are any known issues or workarounds and if not, we can look at the amount of data to see if it's something that could be reduced.

For example, steps that are importing non-Windows file evidence from the FNMS inventory source can be reduced by excluding file evidence directories.

So to fully help move this forward, we need to know which steps in the log are taking the longest time.

(Anything expressed here is my own view and not necessarily that of my employer, Flexera)
If the solution provided has helped, please mark it as such as this helps everyone to know what works.

SandeepKolhe · ‎Dec 12, 2019

Yes we import SCCM. Also we identified more sessions from SCCM end in the FNMS database which causing blocking/locking and deadlock issues so sometimes our import is not going to happen. But SCCM import is there since long back before upgrade, but post upgrade why it's taking much time?

Also, is there any recommendations to increase cpu, memory or virual platform on DB server to fix this issue. We have upgraded FNMS from 2016 R1 to 2019 R1.

Can you please suggest what changes need to be done if it's with import or any hardware changes on DB server etc.

Thank you.

Sandeep

kclausen · ‎Dec 13, 2019

What version of SQL Server do you have?

Starting with version 2019 R1, Flexera recommends that all of your FlexNet Manager databases be set with a compatibility of "SQL Server 2012" for optimal performance.

If your databases are at a higher level, try to downgrade the compatibility level to 2012.

mrichardson · ‎Dec 13, 2019

To further refine Kirk's point, if you are on SQL Server version 2012 - 2016 then the compatibility level should be at 110.

If you are using SQL Server 2017 with 2019 R1 or above then you should set it either to 110 or 140.

Any other configuration has been known on several occasions to cause poor performance, setting to these will give the best chance at faster import times.

(Anything expressed here is my own view and not necessarily that of my employer, Flexera)
If the solution provided has helped, please mark it as such as this helps everyone to know what works.

SandeepKolhe · ‎Dec 13, 2019

The SQL server version is 2012 sp4 and compatibility level is "SQL Server 2012 (110)".

SandeepKolhe · ‎Dec 13, 2019

Also we have rebuild indexes immediate after upgrade and build statistics as well.

ccrawford · ‎Dec 13, 2019

Hello Sandeep,

I noticed on the image you uploaded to the support case has the FNMSCompliance database Recovery model set to "FULL" which is not recommended as per the System Requirements documentation:

https://helpnet.flexerasoftware.com/fnms2019r1/EN/SysReq/index.html#FNMS_sys_req/RN_sys_req_hardware.html

Please ask your DBA to modify these databases carefully to their recommended "SIMPLE" recovery model and the performance should improve.

“Things are only impossible until they are not.”
― Jean-Luc Picard

SandeepKolhe · ‎Dec 13, 2019

Thank you very much, will check with our DBA and get it modified to "Simple", and will see how will be the performance.

Thank you.

Sandeep

ccrawford · ‎Dec 16, 2019

Hello Sandeep,

Please note that doing so may affect DB monitoring tools and database recovery in event of failure. I would highly recommend consulting your DBA first.

However, if you do run a 'FULL' recovery model you may need to increase the amount of resources you provide SQL server - or you may even have a problem with the transaction log growing too large.

At this point I would recommend opening a support case, I have attached an SQL Script which your DBA can run and provide to the support personnel to diagnose the issue further.

“Things are only impossible until they are not.”
― Jean-Luc Picard

SandeepKolhe · ‎Dec 16, 2019

Thanks for your response.

Our DBA raised concern to modify recovery model to "Simple" , if we modify it to Simple, our rpo will increase to 24 hours from the current 30 minutes. So we can't go ahead to modify Recovery model. We need to see another option how to boost performance.

We have raised support case and shared them DB diagnostic details.

Thanks again for your responses.

ccrawford · ‎Dec 16, 2019

Hello Sandeep,

I'm a bit confused by your DBA's response of saying that 'Simple' recovery models would slow down FNMS. However, it is fine to leave it on 'FULL' but your DBA must perform regular transaction log back ups and shrinking to maintain database performance.

The results you uploaded to the case I have reviewed and they show that the transaction log is filled and awaiting a back up. This is not a problem of FNMS, but a problem caused by database maintenance tasks not being performed. The FNMSCompliance DB transaction log is 300 GB and the max file size is 300 GB - this will most definitely cause issues.

“Things are only impossible until they are not.”
― Jean-Luc Picard

SandeepKolhe · ‎Dec 19, 2019

Hello,

Our DBA suggested not to change recovery model to "Simple" from "Full" as this will affect database recovery in event of failure so management is not agree to change it as this is very critical tool for them.

As per suggestion provided by support, our DBA has took care for transaction log backup, they did necessary changes in the database so hopefully import should complete in minimal time.

Let's see tonight's import how it goes. I will update this thread tomorrow. Thank you very much.

Sandeep