FlexNet Manager for Engineering Applications is performing poorly with ?Declining new events Queue size exceeds the limit? and deadlock errors, why is this?

[#1]

Summary

FNMEA is performing poorly with ?Declining new events Queue size exceeds the limit? and deadlock errors, why is this?

Symptoms

• A combination of one or more of the following error message is present in the log files of FNMEA:

  2018-07-12 16:32:54,449 ERROR [manager.usagedatacollector] [pool-150-thread-1] Declining new events Queue size exceeds the limit = 257626
  

  2018-07-12 14:28:17,188 ERROR [STDERR] Caused by: [Incident# 4971-8819913] com.flexnet.platform.exceptions.FLEXnetOopsException: An unknown error has occurred. Please contact the FLEXnet administrator. [Incident# 4971-8819913]
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at com.flexnet.manager.services.license.LicenseFileProcessor.getUnparsedLineMap(LicenseFileProcessor.java:243)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at com.flexnet.manager.services.license.LicenseFileProcessor.processLines(LicenseFileProcessor.java:173)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	... 13 more
  2018-07-12 14:28:17,188 ERROR [STDERR] Caused by: org.hibernate.exception.LockAcquisitionException: could not initialize a collection: [com.flexnet.manager.entities.LicenseFile.vendorLines#357340]
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:82)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.loader.Loader.loadCollection(Loader.java:2001)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:36)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:565)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:63)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1716)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:344)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at org.hibernate.collection.PersistentSet.iterator(PersistentSet.java:163)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	at com.flexnet.manager.services.license.LicenseFileProcessor.getUnparsedLineMap(LicenseFileProcessor.java:226)
  2018-07-12 14:28:17,188 ERROR [STDERR] 	... 14 more
  2018-07-12 14:28:17,188 ERROR [STDERR] Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 103) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
  

• Slowness of processing your data.
• Numerous tasks showing a status of SUSPENDED from the following query:

  SELECT * 
  FROM sys.dm_exec_requests 
  CROSS APPLY sys.dm_exec_sql_text(sql_handle) 
  SELECT * 
  FROM sys.sysprocesses 
  WHERE open_tran = 1

Cause

This error is indicates that FNMEA is currently under high load and has stopped processing real time usage events, the usage events are used display activity on the "Activity" tab.

Resolution

To resolve the issue, there are several steps that can be tried:

• Assign a higher maximum heap size to the module experiencing this issue (either admin and/or reporting).
• the License server Scheduling for each license server could be reviewed and adjusted from 1 minute to 5 minutes for example. This setting can be accessed from the Configure and Manage License Server page. This value controls the frequency of calls made from the Admin machine to each license server, this is then processed and passed into the activity tab.
• If the Admin and Reporting Module are on the same machine intensive tasks such as running reports could be scheduled at different time to reduce the load on the machine and the available resources.
• The following steps can be tried:
  1. Stop FNMEA servers
  2. Run the following queries via SQL Server Management Studio:

     USE master
     ALTER DATABASE flexnet SET SINGLE_USER
     ALTER DATABASE flexnet SET ALLOW_SNAPSHOT_ISOLATION ON
     ALTER DATABASE flexnet SET READ_COMMITTED_SNAPSHOT ON
     ALTER DATABASE flexnet SET MULTI_USER
     

  3. Start FNMEA servers.
• Assign more swap and / or system memory to the machine hosting the SQL Server and / or admin and reporting services

Additional Information

Additional information on increasing the memory allocated to the Java Heap used by FlexNet Manager can be located in the FlexNet Manager Installation Guide, Chapter 7: Configuring FlexNet Manager for Engineering Applications, Setting the Java Heap Size for FlexNet Manager for Engineering Applications.