Free Disk Space Limitations and Behavior for IDOL Components

This page describes how different IDOL components behave when disk space is low.

IDOL Index Components

Indexing requires both temporary disk space to queue and process the data, and permanent disk space after data is converted into its indexed format. The main IDOL Server indexing components (Content and DIH) all support the configuration parameter MinFreeSpaceMB, which can be considered as the limit at which each component prevents further processing.

When the disk limit is reached, all three components reject new incoming index jobs, returning an INSUFFICIENT DISK SPACE message to the client, rather than an index ID. They accept new jobs only when the free disk space has risen above the limit. The expectation is that the client backs off or queues data while you free enough disk space for processing to continue.

DIH bases the free disk space calculation on the free space available for their configured incoming directory.

The Content component additionally checks free disk space immediately before commencing an index job. If there is insufficient disk space to process that job it fails the index action immediately, reporting a -2 (out of disk) error in the IndexerGetStatus action response. You must resend the job if you want to retry after you have freed disk space.

Content additionally calculates the amount of disk space that is likely required to complete a job, based on the amount of data held in the index cache and the size of the incoming XML/IDX file and with the additional buffer of the configured MinFreeSpaceMB. If the amount of free disk space falls below this limit during processing of an index action, the job is paused. After you free the disk space, you can resume the job by sending the IndexerGetStatus action with the IndexAction parameter set to Restart. The server does not restart the job itself.

If you cannot free disk space without shutting down the server, then you can send the Stop service action (or use the Stop Service feature on the Service Control tab of the Console page in the Control section of IDOL Admin) before restarting the job. This action cancels the index job to stop the server, but the server retries the job on restart.

Content calculates its free disk space for each of the configured data paths (DyntermPath, NodetablePath and so on) individually.

Connectors and CFS

CFS pauses ingestion when the free space on the disk where it is running falls below 1 GB. A background thread checks the space every five seconds. CFS continues to accept requests, but it queues them without processing them. It continues to process any documents in the index queue.

You can configure Connectors with the MinFreeSpaceMB configuration parameter, which has a default value of 1.5 GB. If the free space on the current drive falls below this value, all tasks stop, and it does not start any new tasks. Your particular connector implementation determines how often it checks the free disk space, using the same functionality that checks to determine whether to stop a task. Typically the check occurs between documents.