Time out error on multi tenant implementation

[adrian_ritz1]

Hello,

We have a multi tenant implementation with a lot of tenants, some of them are failing with Timed out error during reconcile, we received the following error:

2022-02-02 08:09:29,025 [INFO ] RecalculateInstalledApplicationsForRemoteACLDevices
2022-02-02 18:09:33,709 [INFO ] Failed to execute Writer 'RecalculateInstalledApplicationsForRemoteACLDevices' from file C:\ProgramData\Flexera Software\Compliance\ImportProcedures\Inventory\\Writer\InstalledSoftware.xml, at step line 1
Error: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
2022-02-02 18:09:33,709 [INFO ] All retries have been attempted for Writer 'RecalculateInstalledApplicationsForRemoteACLDevices'
2022-02-02 18:09:33,724 [ERROR] System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at ManageSoft.Compliance.Importer.Logic.XML.Writer.Execute(IExecutionContext context)
at ManageSoft.Compliance.Importer.Logic.ComplianceImporter.ExecuteWriters(ComplianceReader p_ComplianceReader, IExecutionContext context, String singleConnectionIdentifier)

We set the time out limit to 10 hours, we even extend it to 16 hours, but we receive error that tempdb is full, we even extended tempdb to 700GB, which should be enough and still receive this error. So I suspect some where a loop in the process, but don't know where. Have some one encountered this issues. The strange thing is that this is not impacting all the tenants, only a few of them

[ChrisG]

I've searched through various records and can't find any previous reports of problems like you're describing with the RecalculateInstalledApplicationsForRemoteACLDevices import step. (That is, unless we go back to before some fixes that were included in the 2015 R2 release, and I know you're not using a release that old!)

Unless somebody else here has experienced problems with this same step, this may need to be diagnosed from the bottom up. Using SQL Profiler to identify a particular query in the RecalculateInstalledApplicationsForRemoteACLDevices step that is running for a long time may be a start.

[adrian_ritz1]

Hi Chris,

We are using the latest version of FNMS available, 2021 R1

I will speak with the SQL team to use SQL profiler to see what's happening there.

Thank you for the hint.