Regenerate Data Indexes
Regenerating an IDOL data indexes rebuilds the index of stored information for optimized fields (MatchType fields, NumericType fields, and so on) from the textual document field values held in the server. You can do this to reconcile the index or indexes with the document data if:
-
validation of one or more indexes has failed, indicating that the data is no longer in a consistent state
-
you have changed the server
[FieldProcessing]
or[Properties]
configuration to: -
make an existing field an optimized type.
-
remove an optimized field.
-
set or clear the
NumericIntegerOnly
setting for a numeric type field.
In addition, you might need to regenerate fields if you have indexed ReferenceMemoryMappedType
fields out of order, with some child fields indexed before their parents.
In these cases, the alternative to regenerating is to reindex all documents. Regenerating is generally significantly faster, as only the requested indexes regenerate.
You can regenerate a data index by using one of the following methods:
- The Regenerate*Index configuration parameters. These configuration options rebuild the index when you restart the server.
- The
DREREGENERATE
index action. This option allows you to rebuild the index while the server is running. - The Regenerate feature in IDOL Admin (accessible from the Control section, on the Console page, Service Control tab).
The following tables describe the regeneration options for different indexes and field types.
Index Type | Validation Repair | Update Configuration | Configuration Parameter | DREREGENERATE Type |
---|---|---|---|---|
BitFieldType
|
![]() |
RegenerateBitFieldIndex
|
BitField
|
|
CountType
|
![]() |
RegenerateCountIndex
|
Count
|
|
GeospatialType
|
![]() |
RegenerateGeospatialIndex
|
Geospatial
|
|
MatchType
|
![]() |
RegenerateMatchIndex
|
Match
|
|
NumericDateType
|
![]() |
RegenerateNumericDateIndex
|
NumericDate
|
|
NumericType
|
![]() |
![]() |
RegenerateNumericIndex
|
Numeric
|
ParametricRangeType
|
![]() |
RegenerateParametricIndex
|
Parametric
|
|
ParametricType
|
![]() |
RegenerateParametricIndex
|
Parametric
|
|
ReferenceType
|
![]() |
RegenerateRefIndex
|
RefIndex
|
|
SecurityType
|
![]() |
RegenerateSecIndex
|
SecIndex
|
|
SortType
|
![]() |
RegenerateSortFieldIndex
|
SortField
|
|
Unstemmed index![]() |
![]() |
RegenerateUnstemmedIndex
|
Regenerate an Index by Using DREREGENERATE
The DREREGENERATE
index action allows you to change the field configuration and regenerate the index in one action. You can also use it to repair the Reference and Security indexes after a validation failure.
For example:
DREREGENERATE?FieldProcessingSection=SetMatchFields&AugmentFieldCSVs=*/NewMatchField,*/SpecialMatch&DiminishFieldCSVs=*/ExistingMatchField
By default, DREREGENERATE
automatically detects the configuration changes that you make and regenerates the indexes accordingly. You can also explicitly select the indexes to regenerate. For example:
http://12.3.4.56:20001/DREREGENERATE?&Type=Numeric,RefIndex
For more information, refer to the IDOL Server Administration Guide and the IDOL Server Reference.
When the index action runs, IDOL runs through all stored documents, and loads the values of any fields to regenerate from disk. It inserts these values into new index structures.
When the process is complete, IDOL Server deletes any existing indexes and replaces them with the newly built versions.
Regenerate an Index by Updating the Configuration File
When you regenerate by updating the configuration file, you must restart the server. Regenerating the indexes also results in an increased startup time.
To regenerate a data index
-
Stop IDOL Server.
-
Make any required changes to the
[FieldProcessing]
or[Properties]
configuration sections. -
Set the required
Regenerate*Index
settings toTrue
in the[Server]
section of the configuration file. -
Restart IDOL Server.
At start-up, IDOL runs through all stored documents, and loads the values of any fields to regenerate from disk. It inserts these values into new index structures.
When the process is complete, IDOL Server deletes any existing indexes and replaces them with the newly built versions.
After the server has started, remember to remove or turn off any Regenerate*Index
settings in the configuration file if you do not want the server to regenerate the indexes every time it starts.
Performance
Regenerating the index by using DREREGENERATE
is typically I/O intensive, because Content must reread all the documents in your index. Where it can, Content generates a second data structure in memory, and swaps over the files when it has completed the process. This option might be more memory intensive than using the configuration option, but it means that you can continue to query the existing index while regeneration is in progress.
Regenerating the index by using the Regenerate*Index
configuration options requires a server restart. In addition, regenerating indexes also results in an increased startup time. The server is not available for further indexing or querying until the regeneration process is complete.
NOTE: Regenerating the unstemmed index is an intensive process and generally takes significantly longer than regenerating other indexes (though it is still faster than reindexing the data).
Notes
Consider the following points when you use the regeneration options:
-
The regeneration affects all indexes of the requested type. For example, if you add a field as NumericType in the configuration file, setting
RegenerateNumericIndex
toTrue
regenerates all existing numeric indexes, as well as creating one for the new numeric field. -
The original document content must be stored in IDOL Server.
If you have set
NodeTableStoreContent
toFalse
, you can only regenerate a given index if all the fields in question are also explicitly configured asStoredType
.