Many systems export data in JSON format. This section describes how to ingest JSON into IDOL using NiFi Ingest.
The steps in this section assume that:
To ingest data in JSON format
Add a GetFileSystem processor to your data flow to retrieve the JSON file(s).
*.json
.adv:FlowFileEmbedFiles
to TRUE. For more information about this property, see Advanced Connector Properties.Configure the ConvertJSONToDocuments processor.
Right-click the processor and click Configure.
The Configure Processor dialog box opens.
[JSONParsing]
section of the CFS configuration file. (When configuring the NiFi processor, do not include the [JSONParsing]
section header). For more information about these parameters, refer to the Connector Framework Server Reference.Connect the "extracted" relationship of the ConvertJSONToDocuments processor to your ingestion pipeline.
TIP: After they are processed, the original FlowFiles that were routed to the ConvertJSONToDocuments processor are routed to the "processed" relationship.
To avoid indexing documents representing the original JSON files, you could auto-terminate this relationship. However, if you are using a document registry service to ensure that documents are indexed in the correct sequence, route the "processed" relationship to an UnregisterDocument processor. For more information about the document registry service, see Index Documents in the Correct Sequence.