PutKafka
A processor that accepts IDOL document FlowFiles, and sends the data to Apache Kafka.
Properties
Name | Default Value | Description | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
IDOL License Service |
An IdolLicenseServiceImpl that provides a way to communicate with an IDOL License Server. |
|||||||||||||
Document Registry Service | A DocumentRegistryServiceImpl that manages and updates a document registry database. | |||||||||||||
Indexer Batch Size | 100 |
The maximum number of FlowFiles to accumulate before indexing documents. Indexing documents in larger batches can be more efficient; indexing smaller batches more frequently means that an individual document might spend less time in the queue. Documents might be indexed in incomplete batches if the Indexing Interval Duration is reached first. |
||||||||||||
Indexing Interval Duration | 5 min |
The maximum amount of time to wait between indexing batches of documents. Documents might be indexed more frequently than this if the Indexer Batch Size is reached first. Waiting longer and indexing documents in larger batches can be more efficient; indexing smaller batches more frequently means that an individual document might spend less time in the queue. |
||||||||||||
Brokers | A comma-separated list of Kafka brokers to send data to. Specify each broker in the format host:port . |
|||||||||||||
Topic | The name of the topic to append messages to. | |||||||||||||
Partition | The partition number to use for distribution of messages. The default is to use a round-robin method of distribution. | |||||||||||||
Producer Option Names |
You can use these properties to configure producer options. The options that you can set are described in the documentation for the librdkafka library. If you want to set these properties, Micro Focus recommends using the guided setup wizard (click ADVANCED and follow the instructions on the GUIDED SETUP tab). |
|||||||||||||
Producer Option Values | ||||||||||||||
Key Field Path | If not set, the processor uses the document reference. | The path of the field in the IDOL document metadata to use as the Kafka message key. You can select a sub-field by using slash-separated field names such as FieldName/SubFieldName . |
||||||||||||
Value Field Path | If not set, the processor uses the document content. | The path of the field in the IDOL document metadata to use as the Kafka message body. You can select a sub-field by using slash-separated field names such as FieldName/SubFieldName . |
||||||||||||
Timestamp Field Path | DREDATE | The path of the field in the IDOL document metadata to use as the Kafka message timestamp. You can select a sub-field by using slash-separated field names such as FieldName/SubFieldName . |
||||||||||||
Timestamp Field Format | EPOCHSECONDS | The date format used in the field specified by "Timestamp Field Path". Specify the format using a standard IDOL date format string. | ||||||||||||
Metadata Mapping Mode | Full |
Specifies how to convert document metadata into Kafka message headers:
|
||||||||||||
Header Field Names |
You can use these properties to map IDOL document fields to Kafka message headers. If you want to set these properties, Micro Focus recommends using the guided setup wizard (click ADVANCED and follow the instructions on the GUIDED SETUP tab).
Example header names and field paths
|
|||||||||||||
Header Field Paths |
Relationships
Name | Description |
---|---|
success | FlowFiles that were indexed successfully. |
failure | FlowFiles that were not indexed successfully. |