Repair broken ESI Indexproxy service [Rebuild Elasticsearch Index]

[#1]

Description

You might encounter errors similar to the below messages seen in the server.log file on the server hosting your ESI Indexproxy which is fundamental for running a Spider Incident/Change environments:

...marking and sending shard failed due to [failed recovery]...
...IndexShardRecoveryException[failed to recovery from gateway]...
...EngineCreationFailureException[failed to create engine]...
...NoSuchFileException[<Drive\ESIdataFolder>\indexproxy\nodes\0\indices\mod_incident_xxxxxx.... 

This problem means that the Elasticsearch index is corrupted (deleted) and it must be rebuilt. 

Replication Scenario

For example, you might try deleting some files in the translog directory of the server to break the Indexproxy service, intentionally or unintentionally during a server cleanup:

<Drive\ESIdataFolder>\indexproxy\nodes\0\indices\mod_incident_xxxxx-xxxxxx-xxxxxx-xxxx-xxxxx\0\translog\translog-XXX.ckp

Resolution

Please run through the following steps to rebuild the index:

1. Shutdown main application(s) that are using the index, as follows:
   a. Spider Incident
   b. Spider Knowledge Base
   c. Spider LCM (Asset/Licensee/Contract)
   
2.  Stop the main ESI service [e.g. net stop indexproxy]
3. Backup (copy/rename) index data folder. [e.g. <Drive\ESIdataFolder>\indexproxy”]
4. Delete index data folder <Drive\ESIdataFolder>\indexproxy”
5. Make sure there is enough free disk space for the rebuilt index ( >= 10 GB depending on the application data size).
6. Restart ESI service and wait 1-2 minutes until properly startup [e.g. net start indexproxy]
7. Restart the main applications.
8. The index should be automatically rebuilt (monitor the above data folder).

This will take some time depending on the size of the application data. Please be patient with the process until it is complete. Once completed, you can start the Indexproxy service non-problematically. 

net start indexproxy