The Index Process
After all the pre-index processing, IDOL Server indexes the documents. This section describes the processes that IDOL Server performs when it indexes your content.
Index Action Process
See also: The Index Process in a Distributed System
The following diagram describes how the IDOL Server Content component processes an incoming index action. The colored boxes correspond to the status that the IndexerGetStatus
action returns for that stage of the process:
For more information about IndexerGetStatus
and the status messages that it returns, refer to the IDOL Server Reference.
Index Process for the IDOL Server Content Component
The following notes provide some additional information:
-
IDOL Server rejects an index action if it is not from an accepted index client, or if it does not have enough disk space to process the operation.
- As an alternative to indexing data by submitting index actions, you can also use the wizard on the Index tab on the Console page in the Control section of IDOL Admin to submit data for IDOL Server to index. You can also use the wizard on the Replace tab on the Console page in the Control section of IDOL Admin to change field values or delete fields from indexed documents.
-
You can use the
IndexerGetStatus
action to pause, and restart the current index action by adding theIndexAction
parameter (set toPause
orRestart
). You can pause only theDREADD
,DREADDDATA
,DREREPLACE
,DRECOMPACT
, andDRETAGDOCCLUSTERS
index actions. -
You can use the
IndexerGetStatus
action to cancel an index action by adding theIndexAction
parameter (set toCancel
). You can cancel any queued index action, which means that IDOL Server does not start to process that operation. When you cancel the currently processing index action, where possible IDOL Server stops processing the operation and reverts any changes. -
You can view the index queue and associated information on the Index Queue tab of the Status page in IDOL Admin.
-
The indexing thread automatically pauses the current operation if there is insufficient disk space.
-
The
DREFLUSHANDPAUSE
index action automatically pauses when the flush is complete. -
The
DREEXPORTREMOTE
operation withBlocking
set toFalse
gives an index status of -38 "Processing in remote engine" until the target server indexes and syncs the exported data, or fails. -
The
DRESHUTDOWN
index action causes the server to flush and exit as soon as the current index operation is complete.
Document Processing
The following process describes the stages of document processing that IDOL Server performs on documents, and the order in which these occur. Some of the steps might not be performed, according to your configuration.
-
IDOL Server determines whether to index the document at all. This step depends on your distribution settings.
-
It identifies basic metadata:
-
Date
-
Database
-
SecurityType
-
AutnRank
-
LanguageType
It processes these according to your field processing rules.
-
-
It runs Automatic Language Detection on data extracted from the configured LangDetectType fields.
-
IDOL Server converts the entire document internally to a UTF-8 representation, according to the encoding information from It runs Automatic Language Detection on data extracted from the configured LangDetectType fields. (if required).
-
It processes the reference fields, and removes duplicate documents according to your deduplication settings.
If you have set up your deduplication process to preserve fields, IDOL Server processes these and adds the preserved fields to the document.
-
IDOL Server processes the conceptual information from the index fields:
-
Stop word removal.
-
It stores information for each term:
-
the stem of the term
-
positional information (used for proximity searches)
-
case
-
unstemmed form
-
AdvancedSearch, AdvancedCaseSearch, and AdvancedPlus functions
-
-
Advanced processing, including Soundex, synonyms, and proper names.
This information is initially added to the index cache and made available when the cache is flushed. Flushing occurs after every index action if you have set
DelayedSync
toFalse
in your configuration file. Otherwise, IDOL Server flushes the cache periodically, as determined by yourMaxSyncDelay
configuration parameter. You can also flush the index cache to disk by using the Sync function on the Service Control tab of the Console page in the Control section of IDOL Admin. -
IDOL Server processes the other field and subindex types (numeric, parametric, match, sort, security, and so on).
-
IDOL Server stores the document content.