Storage of Document Content
See Also: Disk Usage in IDOL
You can reduce the amount of storage space that IDOL requires by not storing the majority of the document content in IDOL. Some functionality is unavailable for non-stored content, but in some cases there are alternative methods that you can use.
There are two methods you can use in IDOL to disable storage of document content:
-
MustHaveCSVs
andCantHaveFieldCSVs
. This method removes fields before the server processes them at all. -
NodetableStoreContent
andStoredType
. In this method, IDOL processes the fields as normal, but does not store the content. In particular, theDRECONTENT
field is usually the largest index field, and it should be processed for terms but not always stored.
The following article describes the functionality that is restricted when you do not store document content.
Summarization
You cannot use any of the query time summary options for fields that are not stored (the Summary
parameter for the query actions, set to Context
, Concept
, and so on).
As an alternative to query time summarization, you can generate a quick summary as part of the ingest process, and store it in a stored field so that is available in queries.
If the text from the original document is available, then you can use the Summarize
action.
Highlight
You cannot use any of the query time highlight options for fields that are not stored (the Highlight
parameter for the query actions, set to Term
, SummaryTerms
, and so on).
You can send the text from the original document by using the Highlight
action (with or without the TextParse
parameter depending on if you have the text or the IDX/XML available elsewhere).
You can also use the View Server View
action for highlighting the original data file formats.
AdvancedSuggest
You cannot use the AdvancedSuggest
configuration option if you do not store the source fields.
Query Summary
The QuerySummary
options do not generally give good results (or might not function at all) if you do not store the source fields (the DRECONTENT
field is usually a source field).
DREREPLACE
A DREREPLACE
index action to replace an index field causes an internal reindex of the document, which might mean that data is lost from the document.
You can still use DREREPLACE
to add or remove values from optimized fields (MatchType, NumericType, and NumericDateType). For example, you can use DREDELETEFIELDVALUE
/DREFIELDVALUE
on NumericType
fields and the corresponding search operators respect the changes. You cannot use DREREPLACE
to add or remove values from non-stored ParametricType fields.
NOTE: Replacing on an optimized field assumes that a value can appear only once in a document.
TIP: As an alternative to the DREREPLACE
index action, you can also use the wizard on the Replace tab on the Console page in the Control section of IDOL Admin to change field values or delete fields from indexed documents.
DREEXPORTIDX and DREEXPORTXML
The DREEXPORT
index actions export only the data that is stored.
FieldText
Optimized FieldText
operations on optimized field types work as normal. However, other field operations do not work.
You can use the EQUAL
operator on a NumericType
field, but you cannot use MATCH
on a NumericType
field. Similarly, you can use MATCH
on a MatchType
field, but not EQUAL
.
Sort
You can only perform Sort
operations on optimized field types.