Monitoring the search index

After an ingestion operation, i2 Analyze updates its search index to reflect the changes to the Information Store. You can monitor the progress of this process, and intervene if the server encounters problems.

Successful ingestion doesn't always mean that indexing will also be successful. There are some circumstances under which malformed data can make it fail. If indexing does fail, however, you don't have to re-ingest all the data. Rather, you can locate the batch that caused the problem and correct the issue there.

i2 Analyze provides three complementary ways of monitoring the state of the search index:

REST endpoint

GET /api/v1/admin/indexes/status provides information about the health of all the indexes that i2 Analyze maintains. The name of the search index is main_index, and when it's healthy the response from the endpoint includes the following output:

{
  "status": {
    ...
    "main": [
      {
        "name": "main_index",
        "populationPaused": false,
        "populationProgress": 100,
        "state": "LIVE",
        "statusMessage": "The Solr collection is healthy. The minimum replication factor can be achieved.",
        "hasFailedBatches": false
      }
    ],
    ...
  }
}

Indexing log file
i2 Analyze outputs messages about the search index to the log file at wlp\usr\servers\opal-server\logs\opal-services\main_index\i2_Indexer.log.
Database view
The Information Store database includes a public view named IS_Public.Indexing_Failed_Batches that contains information about problems with the indexing process. It can look like this:
batch_id
time_stamp
schema_type_id
error_message
50
2023-07-25 14:56:09.919000
ET1
...
52
2023-07-25 14:56:20.147000
ET5
...

batch_id	time_stamp	schema_type_id	error_message
50	2023-07-25 14:56:09.919000	ET1	...
52	2023-07-25 14:56:20.147000	ET5	...

Detecting problems

To monitor the search index for problems that occur after data ingestion - and to find the cause of any such problems - you can use these tools together:

Poll the indexes/status endpoint periodically. In particular, watch out for the main index's populationProgress falling below 100 for an extended time, and for hasFailedBatches being true.
Alternatively or additionally, check the i2_Indexer.log file for any messages that begin with "Failed to index the batch...", like this one:
```
2023-07-25 15:56:21,943 WARN  SolrIndexSubscriber - Failed to index the batch with ID '52'. Fix and re-ingest the data, and then use the admin/indexes/retryfailed REST endpoint to try again.
```
The batch ID in the log file corresponds to the batch_id column in the database view.
Use the information in the Indexing_Failed_Batches database view to locate the incoming data that caused indexing to fail.

Resolving problems

To restore the search index to full health, you must fix the faulty data and then re-ingest and re-index it.

Correct the problem with the incoming data according to the information in the error messages
Use your ETL pipeline to re-ingest the batch that caused the problem.
Call a second REST endpoint to retry indexing of the batch that failed on the previous attempt.
The POST /api/v1/admin/indexes/retryfailed endpoint adds all failed batches to the queue for indexing.
After you call the endpoint, continue to monitor the indexing process as described above.

Note: Both REST endpoints require the logged-in user to have administrator permissions in order to call them successfully.