cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Reconcile Pending for past 5 days

ImIronMan
By Level 6 Flexeran
Level 6 Flexeran

Hi Team,

The reconciliation which started on 25th of May is still not completed and gave error summary as  "This task is marked as ‘Timed out’ because it did not complete in the expected time" while status shows "Pending".

However I can see the services are running properly and in task scheduler as well (on app server) I see the "import inventories and reconcile" task showing "ready" with out any problem

Also I cant see the "reconcile" button enabled. Attached the screenshot for your ref. Can you please help on urgent basis, how can I get back the reconcile button and start it

 

Regards

Rajesh Ponnala

(7) Replies
ImIronMan
By Level 6 Flexeran
Level 6 Flexeran

Also, a new "inventory import" task started yesterday morning 06.00 am to import SCCM data but is never complete. It is still in "Pending" status.

 

@DAWN 

@mgunnels1 

@mfranz 

If you receive no error during reconcile, probably are other tasks that are hanged or lost, and this is messing up with your reconcile.

Double check if Flexera services are started.

Flexnet Manager Suite batch process scheduler

Flexnet Manager Suite batch processor

 

Also you can check the flexera batch processor queue to see if there are lost tasks.

There should be <Flexera Install directory>\Flexera Software\Flexnet Manager Platform\dotnet\bin

There you should have batchprocesstaskconsole.exe

run this like batchprocesstaskconsole.exe list-tasks, the list should be empty or at least you can see which tasks are causing trouble.

 

@adrian_ritz  Thank you for your response.

I have checked already the services and tasks for FNMS all are running with out any issues.

Once I run the batchprocessconsole lists-tasks, I can see

"Inventory Import" tasks were started on 25-05-2019 and still in process, as per attached screenshot. Can you please have a look at it and suggest  how can we release these ?

Regards

Rajesh Ponnala

According to Flexera documentation:FlexNet Manager Suite System Reference.pdf and please be aware that Flexera recommend this only as last resort and when you are sure the task is lost:

Killing or Failing Tasks
There is no direct way to kill tasks in the batch processing service. You may be able to interrupt and close
executables called by the batch processor.
You can also 'fail' tasks; but misusing this feature in an attempt to interrupt and kill the task poses a
considerable risk to system stability and data integrity. This is because failing the task only instructs the batch
scheduler process to regard the task as dead, but does not stop the batch processor from running the task.
A task should only be failed when it is 'lost'. This term means that execution has finished in the batch
processor, but the batch scheduler has not been notified of its completion. If the task imposes constraints
(listed in Batch Processing Constraints), its coninued presence in the batch scheduler list may be blocking other
tasks. In the very rare case of a suspected lost task, first validate that the batch processor is (still) configured
properly to send messages to the batch scheduler; and then you can validate that a task is lost, and if
necessary fail it, using these steps:
1. On your batch server, inspect the batch processor log files under %ProgramData%\Flexera Software\
Compliance\Logging\BatchProcessScheduler.
2. If there are any error messages indicating that the batch processor cannot send messages to the batch
scheduler, suspend this process, and first remedy the configuration of your batch scheduler and batch
processor.
Particularly for server identification issues, you may wish to re-run the PowerShell configuration scripts
provided for installation and implementation, as these set up the correct relationship between the two
services. You may alternatively need to repair permissions on the message queue; ensure MSMQ is
running on all applicable servers; or take other action as required to remedy the particular problem.
When these two services are again communicating properly, you may resume stepping through this
process to clean up any task you suspect was lost during the period of messaging failure.
3. Execute:
ShadowHost.exe BatchProcessTaskConsole.exe list-tasks
The output includes the message GUID, which can be used when searching the batch scheduler/processor
logs. Depending on when the breakdown occurred in status messaging from the batch processor back to
Server Scheduling the batch scheduler, the suspect task may be shown as either Queued (if the batch processor hasn't yet
picked up the task, or if the message that the batch processor started work did not get through) or
Processing (if only the end-of-task success or failure message was lost). Any other value of the task
status means that the task is not lost, and you should exit this process and review alternative explanations.
4. Re-examine the batch processor log file, now looking for evidence that the suspect task has indeed
finished. Alternatively, in cases where the batch processor starts a child process, examine that process or
its logs for evidence of completion.
The underlying executables that are run in the child processes are listed in Batch Scheduler Command
Line.
5. When the log files show the suspect task has definitely completed (successfully or otherwise, with mere
completion being the requirement), and the list-tasks output still shows the same task as either
Queued or Processing, copy the message ID from the output of the list-tasks command.
6. Substituting your copied message ID for the placeholder shown below, execute:
ShadowHost.exe BatchProcessTaskConsole.exe fail-task -m messageGUID
The batch scheduler updates the status of the task to Error to mark the failure (you can verify this by rerunning
the list-tasks command shown earlier).
Any blocked tasks can now be queued by the batch scheduler, and then executed by the batch processor.

Recently, I used the instructions provided by adrian_ritz1 successfully to kill a hung task.
After I killed the task I still couldn't get subsequent reconciliations to run. I had to find and delete the records for the hung jobs (2018 R2 Release). After that, scheduled reconciliation tasks started running correctly.

You should not need to do this but if you find yourself in the same place here are the queries I used.

SELECT t.DefaultValue,*
FROM [dbo].[BatchProcessExecution] b
INNER JOIN BatchProcessType t
ON b.BatchProcessTypeID = t.BatchProcessTypeID
ORDER BY BatchProcessExecutionID DESC

-- Use this process to delete old entries. Especially necessary after produciton database refresh.
Delete FROM [dbo].[BatchProcessExecution] WHERE Submitted < SYSDATETIME() AND BatchProcessExecution.Progress = 0

Hi,

BatchProcessTaskConsole.exe list-tasks
This command is equivalent to the following database query (which you could also use within Microsoft SQL
Server Management Studio):
SELECT *FROM BatchProcessExecutionInfo bi
WHERE BatchProcessStatusID IN (1, 2, 3)
ORDER BY bi.StatusOrder, bi.DateOrder

So if you kill the task with BatchProcessTaskConsole.exe, the tasks should be also deleted in the database. This will not work if Flexera services are stopped.

I also encountered a case, where the logs where fine, finished with no error, but the task in the database was with status 3 (processing), I assume that do to an error, the status came back in reverse order and tasks was lost. Having this status the next tasks will no start, because the tasks have some dependencies and will not work in parallel. But if I killed the task with BatchProcessTaskConsole.exe the tasks was automatically updated to status 4 in database and the remaining tasks started to be processed. To kill a tasks in this way, you need to be sure that the tasks is lost, that's why you need to check the log files before using this command.

Hi Rajesh,

After stopping the process by adrian_ritz1 his instructions the process should start again.

When the issue happens again you might want to verify the ComplianceReader logging folder on the app server and check the log file (C:\ProgramData\Flexera Software\Compliance\Logging\ComplianceReader). You should see the step here where it goes wrong. If there is a timeout error like "Error: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding." the best thing to do is extending the timeout period.

For extending the timeout period do the following:

Create a new Registry StringValue in:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ManageSoft Corp\ManageSoft\Compliance\CurrentVersion

Name: CommandTimeout

Value: 432000

This value will change the timeout from 2 hours to 120 hours, allowing for larger amounts of data to be processed. (this is just an example value)

hope this helps

Stefan