DIH Distribution Mode Features

This section describes the features that are or are not available for different DIH distribution modes, and describes the data volumes that you can expect in each distribution mode. You can use the two tables on this page to help decide which distribution mode you want to use.

Features Available in DIH Distribution Modes

The following table describes the features that are available in various DIH distribution modes.

Feature Simple Distribute Simple Distribute with DistributeSendMinimal Batch Mode DistributeByReference (hash) DistributeByFields (hash) Consistent HashingNote Round Robin
KillDuplicates Note
KillDuplicates on any field Note
KillDuplicates with KeepExisting Note
Can use PreserveDREADD or ConvertToDREADD
Quick document parsing (IDX only) N/A N/A Note
Can distribute DREREPLACE data Note Note
Change the number of childrenNote Note
Automatic redistribution when adding engines
Automatic redistribution when removing engines
Redistribute when children are down Note
Weighted children
Update-only children
Respect child fullness
Take children up or down for query
Clear data in oldest child

Data Transfer Volumes for DIH Distribution Modes

The following table describes the data transfer volumes for the different DIH distribution modes in terms of the fraction of the number of documents in the incoming IDX or XML. These approximately correspond to data file size (ignoring marginal overheads such as XML headers). However, if your documents differ widely in size or you index in small batches, these values will only emerge as long-term averages.

In the table:

  • N is the number of server groups

  • M is the total number of child servers

  • r is the number of replicas configured in Consistent Hash mode

Amount of Data Simple Distribute Simple Distribute with ConvertToDREADD Simple Distribute with DistributeSendMinimal Batch Mode Hash-Based Modes Hash-Based Modes with Consistent Hashing Round Robin Round Robin with RespectDocumentDate
Saved to incoming 1 1 1 1 1 1 1 1
Read by the DIH (parsing) - - 1 - 1 1 - 1
Written by the DIH (parsing) - - 1Note - M/N (r+1)M/NNote - M/N
Total read or sent for indexing to child servers M - M/NNote M/N M/N (r+1)M/NNote M/N M/N
Volume received per child for one job 1 - 1/NNote 0 or 1Note 1/N (r+1)NNote 0 or 1 0 or 1Note
Long term average value per child 1 - 1/NNote 1/N 1/N (r+1)NNote 1/N 1/N