When you instruct the IDOL Content component to eliminate duplicate document copies at index time using a specific ReferenceType
field (by setting the KillDuplicates
parameter in the IDOL Content component configuration file), it automatically uses any field listed for PropertyFieldCSVs
alongside this ReferenceType
field in the IDOL Content component configuration to eliminate duplicate document copies as well.
However, Content cannot use the same field for deduplication as for the Combine
action parameter, because the Combine
operation clashes (carried out at query time) with Content eliminating duplicate fields. This clash means that, if you want to eliminate duplicate document copies and use the Combine
action parameter, you must set up separate ReferenceType
fields for these processes.
Open the IDOL Content component configuration file in a text editor.
In the [FieldProcessing]
section, add two processes that identify ReferenceType
fields (note that you must set up a field process to identify ReferenceType
fields before you start indexing documents into Content). One of them is used to eliminate duplicate copies of documents, and the other one is used for the Combine
operation.
For example:
[FieldProcessing] 0=MyFirstProcess 1=MySecondProcess 2=SetUpReferenceFields 3=SetUpMoreReferenceFields
Create a section for the processes that you added, and in each section, create a property for the respective process (you define the property later by setting one or more applicable configuration parameters). Identify the fields that you want to associate with each process.
NOTE: The properties that you create must not have the same name as the processes.
For example:
[MyFirstProcess] Property=MyFirstProperty PropertyFieldCSVs=*/MyField,*/MySecondField [MySecondProcess] Property=MySecondProperty PropertyFieldCSVs=*/MyThirdField [SetUpReferenceFields] Property=ReferenceFields PropertyFieldCSVs=*/DREREFERENCE,*/URL [SetUpMoreReferenceFields] Property=MoreReferenceFields PropertyFieldCSVs=*/DRETITLE
Create a section for each of the properties and specify appropriate configuration settings for each. These configuration parameters define the processes that are applied to all the fields (or all documents that contain the fields) that you previously associated with the processes. For example:
[MyFirstProperty] HiddenType=True [MySecondProperty] Index=True [ReferenceFields] ReferenceType=True TrimSpaces=True [MoreReferenceFields] ReferenceType=True TrimSpaces=True
Save and close the configuration file.
Restart the IDOL Content component for your changes to take effect.
After you index documents into Content, you can use, for example, the */DREREFERENCE
field to eliminate duplicate copies of documents. (Content then automatically also uses the */URL
field for deduplication because it is listed alongside */DREREFERENCE
for PropertyFieldCSVs
.) This leaves you free to use the */DRETITLE
field for the Combine
operation.