DIH Distribution Mode Features
This section describes the features that are or are not available for different DIH distribution modes, and describes the data volumes that you can expect in each distribution mode. You can use the two tables on this page to help decide which distribution mode you want to use.
Features Available in DIH Distribution Modes
The following table describes the features that are available in various DIH distribution modes.
Feature | Simple Distribute | Simple Distribute with DistributeSendMinimal | Batch Mode | DistributeByReference (hash) | DistributeByFields (hash) | Consistent HashingNote | Round Robin |
---|---|---|---|---|---|---|---|
KillDuplicates
|
|
|
|
|
|
|
|
KillDuplicates on any field |
|
|
|
|
|
|
|
KillDuplicates with KeepExisting |
|
|
|
|
|
|
|
Can use PreserveDREADD or ConvertToDREADD |
|
|
|
|
|
|
|
Quick document parsing (IDX only) | N/A |
|
N/A |
|
|
|
|
Can distribute DREREPLACE data |
|
|
|
|
|
|
|
Change the number of childrenNote |
|
|
|
|
|
|
|
Automatic redistribution when adding engines |
|
|
|
|
|
|
|
Automatic redistribution when removing engines |
|
|
|
|
|
|
|
Redistribute when children are down |
|
|
|
|
|
|
|
Weighted children |
|
|
|
|
|
|
|
Update-only children |
|
|
|
|
|
|
|
Respect child fullness |
|
|
|
|
|
|
|
Take children up or down for query |
|
|
|
|
|
|
|
Clear data in oldest child |
|
|
|
|
|
|
|
Data Transfer Volumes for DIH Distribution Modes
The following table describes the data transfer volumes for the different DIH distribution modes in terms of the fraction of the number of documents in the incoming IDX or XML. These approximately correspond to data file size (ignoring marginal overheads such as XML headers). However, if your documents differ widely in size or you index in small batches, these values will only emerge as long-term averages.
In the table:
-
N is the number of server groups
-
M is the total number of child servers
-
r is the number of replicas configured in Consistent Hash mode
Amount of Data | Simple Distribute | Simple Distribute with ConvertToDREADD | Simple Distribute with DistributeSendMinimal | Batch Mode | Hash-Based Modes | Hash-Based Modes with Consistent Hashing | Round Robin | Round Robin with RespectDocumentDate |
---|---|---|---|---|---|---|---|---|
Saved to incoming | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Read by the DIH (parsing) | - | - | 1 | - | 1 | 1 | - | 1 |
Written by the DIH (parsing) | - | - | 1Note | - | M/N | (r+1)M/NNote | - | M/N |
Total read or sent for indexing to child servers | M | - | M/NNote | M/N | M/N | (r+1)M/NNote | M/N | M/N |
Volume received per child for one job | 1 | - | 1/NNote | 0 or 1Note | 1/N | (r+1)NNote | 0 or 1 | 0 or 1Note |
Long term average value per child | 1 | - | 1/NNote | 1/N | 1/N | (r+1)NNote | 1/N | 1/N |