Advanced Connector Properties

IDOL NiFi Ingest Connectors support some properties which you can use to customize the behavior of the processor. These properties are supported by all connectors.

Property Name Description Default
adv:AllowedClusterNodesRegex A regular expression that matches the host names of the NiFi instances to run the connector on (if you have a cluster of NiFi instances). Run on all nodes of the cluster.
adv:ConnectionValidation A Boolean value that specifies whether the connector validates its connection to the repository using any connection details or credentials you have provided. When this feature is enabled the connector provides a warning if the connection is not successful. True
adv:EnableDebugStatistics A Boolean value that specifies whether to collect additional statistics about the number and frequency of API calls that are being sent to the repository. Some connectors can store these statistics, to help you monitor the requests being made. Some connectors do not store any additional statistics. See also StatisticsIntervals. False
adv:EnableReport A Boolean value that specifies whether to enable the "report" output relationship. When you set this property to TRUE, the connector can send FlowFiles (each reporting a single error) to this relationship. For more information, see Generate Error Reports. False
adv:FlowFileEmbedFiles

A Boolean value that specifies whether to include binary file content in FlowFiles, rather than including only file paths. For example, if you set this property to TRUE for a File System Connector, a FlowFile created by the connector can include a file that was retrieved from the file system.

Setting this property to TRUE can reduce performance and is not necessary if you are using a single NiFi instance. If you run a NiFi cluster, the files retrieved by a connector must be available to all of the nodes in the cluster. You can either configure an ingest shared path (by setting the property adv:IngestSharedPath), or embed the files in the FlowFiles (by setting this property to TRUE).

False
adv:IngestBatchSize

The number of documents to create before releasing a batch of documents to the outgoing queue.

This property only applies to processors, such as GetFileSystem, that perform the synchronize action and have names beginning with "Get".

100
adv:IngestSharedPath

The directory in which the connector saves files that were downloaded from the data repository. You can specify the same directory for multiple connectors, because each processor creates a subfolder within the specified path.

If you run a NiFi cluster, the files retrieved by a connector must be available to all of the nodes in the cluster. You can either configure an ingest shared path (by setting this property to a directory that is accessible from every node in the cluster), or embed the files in the FlowFiles (by setting adv:FlowFileEmbedFiles).

 
adv:PurgeDatastore

A Boolean value that specifies whether to purge (clear) the datastore after each scheduled synchronize cycle. By default, a connector stores information about the items it has retrieved from a repository so that it does not need to retrieve them again unless they are modified. If you set this property to TRUE the connector performs a full synchronize on every scheduled synchronize cycle.

NOTE: If you set this property to TRUE, the connector cannot issue delete commands for items that were previously synchronized but have been deleted from the source repository.

This property only applies to processors, such as GetFileSystem, that perform the synchronize action and have names beginning with "Get".

False
adv:StatisticsIntervals

The time intervals for which the connector returns statistics. The connector stores statistics, including the number of ingest-adds, ingest-updates, and ingest-deletes that have occurred. You can view these statistics through the advanced configuration UI (see View Connector Statistics).

You can set this parameter to a comma-separated list of one or more of the following values (in seconds):

10,60,300,1800,3600,10800,86400,604800

See also EnableDebugStatistics.

The connector returns statistics for the last 10 seconds, 1 minute, 5 minutes, 30 minutes, 3 hours, 1 day, and 1 week.
adv:TaskMaxDuration The maximum amount of time that the processor should run for. If the time limit is reached the connector stops until the next scheduled task begins. Specify the duration with time units, for example 2 hours or 30 min.  
adv:Threads

The number of threads to use. By default a connector uses up to five threads for synchronization. The connector might be able to synchronize with the repository in less time if you increase the value, but this places additional load on the repository because it must respond to more frequent requests.

This property applies to all types of connector processors.

5
ATTR:name Adds a FlowFile attribute, with the specified name, to every FlowFile produced by the connector.  
META:fieldname Adds a metadata field to every document produced by the connector. The field has the name fieldname and the value of the property. For example, to add a field named SecurityType to every document, set a property named META:SecurityType.  

To set an advanced property

  1. Stop the connector, if it is running. To do this, right-click the processor and click Stop.
  2. Right-click the processor and click Configure.

    The Configure Processor dialog box opens.

  3. Click the Properties tab.
  4. Click Add .

    The Add Property dialog box opens.

  5. Type the name of the property that you want to add and click OK.
  6. Set a value for the property.
  7. Click APPLY.

To remove an advanced property, find it in the properties list and click the delete icon .