Information Store data ingestion

To enable its analytical capabilities, i2® Analyze places requirements on the data records that it processes. i2 Analyze records must conform to the general data model and a deployment-specific i2 Analyze schema, and they need information that might not be present in external sources. Before you can add data from your source to the Information Store, you must transform it to meet the same requirements.

i2 Analyze provides two mechanisms that make it easier to align your data with an i2 Analyze schema and prepare it for ingestion:

• The i2 Analyze deployment toolkit includes a command for creating staging tables that match the structure of the schema, in the same database as the Information Store. If you can load your data into the staging tables successfully, the Information Store can ingest your data from those tables.

• During data ingestion, i2 Analyze uses ingestion mappings that describe how to create records in the Information Store from the data in the staging tables. You can define these mappings in one or more files.

There are two scenarios for ingesting data from an external source. The first time that you load data of a particular item type from an external source, you perform an initial load. In an initial load, you are presenting the data to the Information Store for the first time. After you perform the initial load, you might want to perform periodic loads that update the Information Store as data is added to the external source or the data is changed. The same staging tables and mapping files can be used in both scenarios.

The following diagram shows the types of data that can be included in each type of load that you might perform. The data in blue is presented to the Information Store for the first time, and the data in red contains updates to data that exists in the Information Store.

The i2 Analyze deployment toolkit provides two import modes that you can use to ingest data:

• Bulk import mode

  The bulk import mode can be used in the following situations:

  • Initial load, new records, no correlation

  • Periodic load, new records, no correlation

  The bulk import mode is significantly faster for ingesting new, uncorrelated records.

• Standard import mode

  The standard import mode can be used in the following situations:

  • Initial load, with or without correlation

  • Periodic load, new and updated records, with or without correlation

  Standard import mode trades some performance for increased data validation and additional operations during ingestion.

  For more information about correlation, see Overview of correlation.

Because ingestion is completed on a per-item-type basis, you can use the bulk import mode for some item types and use standard mode for others from the external source. You can also use the bulk import mode to complete an initial load, and then use the standard import mode to complete periodic loads to update the data.