Using bulk import mode

Bulk import mode can be used for improved ingestion performance when you are populating the Information Store with new records as part of an initial or periodic load.

Before you begin

  • Before you run the ingestion commands, ensure that you complete all of the tasks in Preparing for ingestion.

  • For Db2, the mechanism that bulk import mode uses to load the data from the staging tables to the Information Store can require changes to the database configuration. Before you ingest data, ensure that your database is configured correctly. For more information, see Database configuration for IBM Db2.

  • You can drop and re-create the database indexes during the ingestion process, which can reduce the total time that it takes to ingest data. For more information about when it is beneficial to drop and re-create the database indexes, see Database index management

About this task

When you instruct the Information Store to ingest the data that you loaded into the staging tables, you do it one ingestion mapping (and one staging table) at a time. The i2 Analyze deployment toolkit and the ETL toolkit both have a command for ingesting the data that is associated with a particular ingestion mapping in a particular mapping file. For more information about the running the command and any arguments, see The ingestInformationStoreRecords task.

When you use bulk import mode, take the following points into consideration:

  • For best performance, ensure that there are no analysts using the system during the ingestion process.

  • Do not include correlation identifiers in the data that you ingest by using bulk import mode. If any correlation identifiers are included, the ingestion fails.

  • You cannot run concurrent ingestInformationStoreRecords commands when using bulk import mode.

  • Source references are not ingested when using bulk import mode. To ingest source references, you can use the standard import mode afterwards to update the ingested records with their source references.

Procedure

After you stage your data and create your configuration files, you can instruct the Information Store to ingest your data by using bulk import mode. The procedure to ingest your data is similar to many others in this process. You start with one type or one staging table, and build from there.

  1. If you have decided to drop the database indexes, do so now.

  2. Use the following command to ingest your data.

    setup -t ingestInformationStoreRecords
-p importMappingsFile=ingestion_mapping_file
-p importMappingId=ingestion_mapping_id
-p importLabel=ingestion_label
-p importConfigFile=ingestion_settings_file
-p importMode=BULK

    For more information about the running the command and any arguments, see The ingestInformationStoreRecords task.

  3. If any errors occur, refer to Troubleshooting the ingestion process.

  4. If you need to re-create the database indexes, do so now.