Add KeyView Extraction
The first step in this example ingestion pipeline is KeyView Extraction. Extraction extracts individual files from containers (such as zip files or other types of archives) so that they can be processed by other processors.
To add KeyView Extraction
- Add a funnel, by dragging the funnel icon
from the components toolbar to the canvas.
-
Create a connection between the GetFileSystem connector and the funnel. Hover the mouse over the connector until you see the connection icon -
- and then drag the icon to the funnel.
The Create Connection dialog box opens.
-
In the For Relationships area, select the success check box, because we want to extract documents that were successfully retrieved. Then, click ADD.
The connection appears on the canvas. NiFi automatically adds a queue between the connector and the funnel.
-
Add a processor, by dragging the processor icon
from the components toolbar to the canvas.
The Add Processor dialog box opens.
-
In the Source list, click idol.nifi.
The list of processors is filtered to those that are provided in the NiFi Ingest distribution.
-
Select the KeyViewExtractFiles processor and click ADD.
The processor is added to the canvas.
-
Create a connection from the funnel to the KeyViewExtractFiles processor. Hover the mouse over the funnel until you see the connection icon -
- and then drag the icon to the KeyViewExtractFiles processor.
The Create Connection dialog box opens.
-
Click the Settings tab.
-
Drag the PriorityAttributePrioritizer from the Available Prioritizers area to the Selected Prioritizers area. This configures the queue such that KeyView will process subfiles before extracting new containers. For more information about this configuration, see Extract Subfiles Recursively.
-
Click ADD.
The connection is created.
In its default configuration, the KeyViewExtractFiles processor creates new documents for extracted subfiles, and routes them to the "subfile" relationship. These subfiles might also be containers, for example a zip file can contain another zip file. To extract files recursively, route this relationship back to the funnel as described in the next step.
-
Create a connection from the KeyViewExtractFiles processor to the funnel. Hover the mouse over the processor until you see the connection icon -
- and then drag the icon to the funnel.
The Create Connection dialog box opens.
-
In the For Relationships area, select the subfile check box and click ADD.
The connection appears on the canvas. By default, the queue name ("subfile") matches the relationship(s) that you selected.
-
Right-click the KeyViewExtractFiles processor and click Configure.
The Configure Processor dialog box opens.
- Click the Properties tab.
- In the properties list, set the IDOL License Service property to the name of the IDOL License Service that you created in Create a License Service.
-
In the properties list, set the KeyView Filter Service property to the name of the service that you created in Add a KeyView Service.
To find out more about this processor and the properties that you can set, see the documentation for the KeyViewExtractFiles processor.
- Click Apply.