Unable to start batch process services post

[emtmeta]

We are doing an upgradation of fnms server from 2018r2 to 2020r2 . We have successfully completed the DB upgrade and started the configuration script after installing new version of fnms .

The script failed at starting batch process services, also manual start is failing with following error at batchprocesstask log

I searched at multiple tables in inventory database and i could find the column OrganizationID, but not sure which column or at which table the OrganizationID column is missing as per below log

021-09-23 04:00:04,816 [tchProcessTask.Program] [ERROR] An error has occurred: Exception has been thrown by the target of an invocation. Schema needs to be updated. Contact your system administrator. Sql text: Invalid column name 'OrganizationID'.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> DevExpress.Xpo.DB.Exceptions.SchemaCorrectionNeededException: Schema needs to be updated. Contact your system administrator. Sql text: Invalid column name 'OrganizationID'. ---> System.Data.SqlClient.SqlException: Invalid column name 'OrganizationID'.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at DevExpress.Xpo.DB.ConnectionProviderSql.<>c__DisplayClass6.<InternalGetData>b__4()
at DevExpress.Xpo.Logger.LogManager.Log[T](String category, LogHandler`1 handler, MessageHandler`1 createMessageHandler)
at DevExpress.Xpo.DB.ConnectionProviderSql.SelectDataSimple(Query query, CriteriaOperatorCollection targets, Boolean includeMetadata)
--- End of inner exception stack trace ---
at DevExpress.Xpo.DB.ConnectionProviderSql.SelectDataSimple(Query query, CriteriaOperatorCollection targets, Boolean includeMetadata)
at DevExpress.Xpo.DB.ConnectionProviderSql.SelectData(Query query, CriteriaOperatorCollection targets, Boolean includeMetadata)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at ManageSoft.Database.Xpo.CustomMSSqlConnectionProvider.ProcessSelectData(SelectStatement select)
at DevExpress.Xpo.DB.DataStoreBase.ProcessSelectData(SelectStatement[] selects)
at DevExpress.Xpo.DB.DataStoreSerializedBase.SelectData(SelectStatement[] selects)
at DevExpress.Xpo.DB.ConnectionProviderSql.SelectData(SelectStatement[] selects)
at ManageSoft.Database.Xpo.CustomMSSqlConnectionProvider.DevExpress.Xpo.DB.IDataStore.SelectData(SelectStatement[] p_Statements)
at DevExpress.Xpo.SimpleDataLayer.SelectData(SelectStatement[] selects)
at ManageSoft.Database.Xpo.XpoSession.XpoDataLayer.SelectData(SelectStatement[] p_Selects)
at DevExpress.Xpo.Helpers.ObjectCollectionLoader.InternalLoadData(ObjectsQuery[] queries)
at DevExpress.Xpo.Helpers.ObjectCollectionLoader.LoadObjects(ObjectsQuery[] queries)
at DevExpress.Xpo.SimpleObjectLayer.LoadObjects(Session session, ObjectsQuery[] queries)
at ManageSoft.Database.Xpo.XpoSession.XpoObjectLayer.DevExpress.Xpo.IObjectLayer.LoadObjects(Session p_Session, ObjectsQuery[] p_Queries)
at DevExpress.Xpo.Logger.LogManager.Log[T](String category, LogHandler`1 handler, MessageHandler`1 createMessageHandler)
at DevExpress.Xpo.Session.GetObjects(ObjectsQuery query)
at DevExpress.Xpo.XPQueryBase.GetObjects()
at DevExpress.Xpo.XPQuery`1.GetEnumerator()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Flexera.Broker.Connections.Impl.DefaultTenantActionsData.GetAllTenants(ConnectionType type)
at Flexera.BatchProcessing.BatchProcessTask.RunTaskWriter.GenerateTasksForAllTenants(BatchProcessTypeConfiguration taskConfig, IEnumerable`1 taskProperties, String taskArgs, Func`2 messageHandler)
at Flexera.BatchProcessing.BatchProcessTask.RunTaskWriter.WriteTaskMessages(BatchProcessTypeConfiguration taskConfig, ProgramOptions options, Func`2 messageHandler)
at Flexera.BatchProcessing.BatchProcessTask.RunTaskWriter.ProcessTaskCommand(ProgramOptions options, String taskName)
at CommandLine.Parser.ParseArgumentsStrict(String[] args, Object options, Action`2 onVerbCommand, Action onFail)
at Flexera.BatchProcessing.BatchProcessTask.Program.Main(String[] args)

[ChrisG]

Based on the stack trace I would guess that this failure is due to a OrganizationID column missing from the Tenant table in the broker database. This is likely a column that has been added since the 2018 R2 release.

(FWIW - I think this error message suggests you are using a multi-tenant "cloud" configuration of FlexNet Manager Suite.)

I'm not that familiar with the broker database, but maybe a further step to update the schema in that database needs to be done as part of the upgrade.

[emtmeta]

Hi @ChrisG

Thanks for the quick note .

FYI, this is an FNMS on premise installation .

And nnow my doubt is , why the scripts says column missing where as the database upgrades were succeeded without any error .

I'm also not familiar in handling such broken DBs , any idea how can I perform the suggested schema upgrade.


Also hoping for more feedback from community colleagues who might seen this error before 🙂

Jv

[ChrisG]

To help clarify the specific scenario you're working with:

1. Are you working with a single-tenant install, a multi-tenant install, or a cloud install of FlexNet Manager Suite On-premises?
2. Do you have a broker database, and did you run the mgsDatabaseUpdate.exe process on that database to migrate its schema? (NB. That is "broker", not "broken". :-))
3. Which of your databases have a Tenant table in them, and do all of those tables have a column named OrganizationID?

[emtmeta]

@ChrisG 

1. i'm working with Single Tenant On-Premise installation 
2. If you are referring to App Broker database , no we don't have.
   
   Yeah , we run the mgsdatabaseupdate in all 3 DBs including FNC compliance.
3. I checked tenant table only in Inventory DB when i connected to customer's system , that that table was having organizationid 
   
   But i need to check other databases with tenant table and with organizationid missing in it.

Regards,

junaid vengadan