Does ManageSoftRL use mgsimport.exe?

[mfranz]

Hi,

I am seeing that, while importing inventories via scheduled task, incoming inventories (to be processed directly) are put to BadLogs. These files import just fine when later imported manually.

My assumption is that while the executable (mgsimport.exe) is in use, the website cannot import stuff and considers this an error, hence BadLogs.

Can someone confirm this behavior?

Best regards,

Markward

[ChrisG]

The ManageSoftRL application in IIS does not directly invoke the mgsimport.exe executable. Both mgsimport.exe and the ManageSoftRL IIS application use a common set of DLLs to perform operations to import data from uploaded files. In the case of IIS the file content to be imported is taken from the stream of data provided through an HTTP PUT operation. In the case of mgsimport.exe the file content to be imported is taken from files on the filesystem under C:\ProgramData\Flexera Software\Incoming.

The import processing is capable of processing multiple files in parallel: a running mgsimport.exe process can quite happily import files from the file system at the same time as IIS is importing several files that are being uploaded from child beacons concurrently.

Log files in the C:\ProgramData\Flexera Software\Compliance\Logging\WebResolvers\ directory on the inventory server may give an indication of why files uploaded to IIS fail to be processed and end up in the BadLogs directory.

[mfranz]

Hi Chris,

I do see 3 to 12 MB of logs every day, like this:

[WARN  2021-08-18 11:21:08,679 34263696ms DatabaseLayerBaseSql  ] Connection owning an AppLock closed prematurely. Clearing the connection pool to avoid the possibility of a dangling lock.
[WARN  2021-08-18 11:21:08,679 34263696ms DatabaseLayerBaseSql  ] Connection state changed unexpectedly: AppLock not released before connection Closed (previously was Open)
This indicates that the connection was closed before releasing an AppLock (a possible bug).
(More severe SQL errors and database connectivity loss can cause this message to show; these are not indicative of a bug. If no connectivity issue is observed, a bug MAY be present.)
THIS IS NOT AN EXCEPTION: the following trace describes the current execution location to assist diagnosis only.
There may be an error logged immediately following this one; it may show the reason for this connection close event.
   at System.Data.Common.DbConnection.OnStateChange(StateChangeEventArgs stateChange)
   at System.Data.SqlClient.SqlConnectionFactory.SetInnerConnectionEvent(DbConnection owningObject, DbConnectionInternal to)
   at System.Data.ProviderBase.DbConnectionInternal.CloseConnection(DbConnection owningObject, DbConnectionFactory connectionFactory)
   at System.Data.SqlClient.SqlConnection.Close()
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at ManageSoft.Database.DatabaseCommand.ExecuteNonQuery()
   at ManageSoft.Database.Impl.DatabaseLayerBaseSql.Execute(CommandType p_CommandType, String p_CommandText, DatabaseParameter[] p_Parameters)
   at ManageSoft.Database.Impl.DatabaseTableBase`2.ReplaceObjects(IEnumerable`1 p_OldObjects, TCollection p_NewObjects)
   at ManageSoft.Database.Impl.SoftwareFileTable.ReplaceSubsetByComputerID(Int32 p_ComputerID, SoftwareFileCollection p_Objects)
   at ManageSoft.Inventory.Importer.DatabaseImport.Commit()
   at ManageSoft.Inventory.Importer.Implementation.RootNodeProcessor.HandleEndElement()
   at ManageSoft.Inventory.Importer.InventoryImporter.Import(String importType, Stream inputStream, Boolean isGzipped, String sourceIdentifier, String targetFilename, MGSLicenseTermCheckResult licenseCheckResult, ImportProcessingStats& stats, IDispatcherPreferences preferences)
   at ManageSoft.Import.Dispatcher.runImporter_ActualExecution_invokeImporter(ItemProcessingTask processingTask)
   at ManageSoft.Import.Dispatcher.runImporter_ActualExecution(MgsAsyncResult mgsAsyncResult)
   at ManageSoft.Import.Dispatcher.runImporter(MgsAsyncResult mgsAsyncResult)
   at ManageSoft.Import.Dispatcher.runImporterUsingImpersonation(MgsAsyncResult mgsAsyncResult)
   at ManageSoft.Import.Dispatcher.doRunImporter(MgsAsyncResult mgsAsyncResult)
   at ManageSoft.Import.Dispatcher.doRunImporterAndComplete(Object state)
   at ManageSoft.Import.Dispatcher.doRunImporterAndComplete_SignatureCompatibleWithDelegateNeededForCustomePool(Object state, DateTime dt)
   at ManageSoft.Threading.CustomThreadPool.ThreadWrapper.ThreadProc()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

[ERROR 2021-08-18 11:21:08,679 34263696ms InventoryImporter     ] Exception processing inventory 'system on XXXXXXXX at 20210818T103342 for machine (Full).ndi.gz'
System.Data.SqlClient.SqlException (0x80131904): Cannot continue the execution because the session is in the kill state.
A severe error occurred on the current command.  The results, if any, should be discarded.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at ManageSoft.Database.DatabaseCommand.ExecuteNonQuery()
   at ManageSoft.Database.Impl.DatabaseLayerBaseSql.Execute(CommandType p_CommandType, String p_CommandText, DatabaseParameter[] p_Parameters)
   at ManageSoft.Database.Impl.DatabaseTableBase`2.ReplaceObjects(IEnumerable`1 p_OldObjects, TCollection p_NewObjects)
   at ManageSoft.Database.Impl.SoftwareFileTable.ReplaceSubsetByComputerID(Int32 p_ComputerID, SoftwareFileCollection p_Objects)
   at ManageSoft.Inventory.Importer.DatabaseImport.Commit()
   at ManageSoft.Inventory.Importer.Implementation.RootNodeProcessor.HandleEndElement()
   at ManageSoft.Inventory.Importer.InventoryImporter.Import(String importType, Stream inputStream, Boolean isGzipped, String sourceIdentifier, String targetFilename, MGSLicenseTermCheckResult licenseCheckResult, ImportProcessingStats& stats, IDispatcherPreferences preferences)
ClientConnectionId:a4f8dc35-7990-40ae-85d7-95eeadbb4ad0
Error Number:596,State:1,Class:21

This seems to happen randomly over the day, but also reliably while the scheduled task is used to import those "invaliddata" inventories again. I also ran the mgsimport.exe manually with -e and they all imported without error. I guess I'll need to open a support case.

Best regards,

Markward

[ChrisG]

If I were to guess, this feels like it could be symptom of an overloaded database which is struggling to keep up with all the activity going on. I'd be looking for things like:

• Any other types of errors in the dispatcher logs around the same time - such as timeout or deadlock errors
• Patterns around what time of day these messages appear (although you've already noted it appears to be random over the day) - and whether other activities (such as imports/reconciliations or other batch processes) are going on in the system around that time
• Any indications of errors around the same time appearing in SQL Server ERRORLOG files