PutKafka

A processor that accepts IDOL document FlowFiles, and sends the data to Apache Kafka.

Properties

Name Default Value Description
IDOL License Service  

An IdolLicenseServiceImpl that provides a way to communicate with an IDOL License Server.

Document Registry Service   A DocumentRegistryServiceImpl that manages and updates a document registry database.
Indexer Batch Size 100

The maximum number of FlowFiles to accumulate before indexing documents.

Indexing documents in larger batches can be more efficient; indexing smaller batches more frequently means that an individual document might spend less time in the queue.

Documents might be indexed in incomplete batches if the Indexing Interval Duration is reached first.

Indexing Interval Duration 5 min

The maximum amount of time to wait between indexing batches of documents. Documents might be indexed more frequently than this if the Indexer Batch Size is reached first.

Waiting longer and indexing documents in larger batches can be more efficient; indexing smaller batches more frequently means that an individual document might spend less time in the queue.

Brokers   A comma-separated list of Kafka brokers to send data to. Specify each broker in the format host:port.
Topic   The name of the topic to append messages to.
Partition   The partition number to use for distribution of messages. The default is to use a round-robin method of distribution.
Producer Option Names  

You can use these properties to configure producer options. The options that you can set are described in the documentation for the librdkafka library. If you want to set these properties, Micro Focus recommends using the guided setup wizard (click ADVANCED and follow the instructions on the GUIDED SETUP tab).

Producer Option Values  
Key Field Path If not set, the processor uses the document reference. The path of the field in the IDOL document metadata to use as the Kafka message key. You can select a sub-field by using slash-separated field names such as FieldName/SubFieldName.
Value Field Path If not set, the processor uses the document content. The path of the field in the IDOL document metadata to use as the Kafka message body. You can select a sub-field by using slash-separated field names such as FieldName/SubFieldName.
Timestamp Field Path DREDATE The path of the field in the IDOL document metadata to use as the Kafka message timestamp. You can select a sub-field by using slash-separated field names such as FieldName/SubFieldName.
Timestamp Field Format EPOCHSECONDS The date format used in the field specified by "Timestamp Field Path". Specify the format using a standard IDOL date format string.
Metadata Mapping Mode Full

Specifies how to convert document metadata into Kafka message headers:

  • Full - Document metadata is converted into Kafka message headers. The header names are full XPath-like indexed field paths, for example FIELD1[1]/@attr.
  • Simple - Document metadata is converted into Kafka message headers. The header names are generated without the indexes, for example FIELD1_attr.
  • None - Document metadata is not converted by default. Use "Header Field Names" and "Header Field Paths" instead.
Header Field Names  

You can use these properties to map IDOL document fields to Kafka message headers. If you want to set these properties, Micro Focus recommends using the guided setup wizard (click ADVANCED and follow the instructions on the GUIDED SETUP tab).

  • IDOL document metadata field paths are slash-separated and can reference attributes using the "@" character. For example, to reference an attribute called "name" on a field named "MyField", enter MyField/@name.
  • The header name can contain placeholders, $1, $2, ... $N that correspond to the Nth asterisk wildcard (*) in the corresponding IDOL document metadata field path.
  • An asterisk wildcard (*) that is used in a path can match only a whole field name or attribute name.
Header Field Paths  

Relationships

Name Description
success FlowFiles that were indexed successfully.
failure FlowFiles that were not indexed successfully.