Ingest JSON

Many systems export information in JSON format and CFS has features to help you convert JSON into IDOL documents.

JSON files are parsed as follows:

  • Any document that has the metadata field AUTN_JSON_PARSING_SECTION is parsed using the settings specified in the section of the CFS configuration file that matches the field value.
  • All other JSON documents are parsed using the settings specified in the [JSONParsing] section of the CFS configuration file.

For information about the JSON parsing configuration parameters, refer to the Connector Framework Server Reference.

The CFS JSON reader accepts a stream of JSON document objects:

{"DREREFERENCE": "one"}{"DREREFERENCE": "two"}{"DREREFERENCE": "three"}

Or an array of JSON document objects:

[{"DREREFERENCE": "one"},{"DREREFERENCE": "two"},{"DREREFERENCE": "three"}]

Example

Consider the following JSON:

{
    "ref": "12345abcd",
    "someField": [
        "value1",
        "value2"
    ],
    "otherfield": {
        "field_name": "myfield",
        "@attribute": true,
        "attrs": {
            "attr1": "fish",
            "attr2": "chips",
            "attr3": "peas"
        },
        "field_value": {
            "attr_name": "key",
            "attr_value": "value",
            "interesting": 7
        },
        "someField": [
            12.9
        ]
    },
    "content_stuff": {
        "1": "page1",
        "2": [
            "page2",
            "page3"
        ]
    }
}

To ingest this JSON, you might use the following configuration:

[JSONParsing]
ReferencePath=ref
ContentPaths=content_stuff/^.*$
FieldSections=SomeField,OtherField
ExpandArrays=true

[SomeField]
Path=someField/^.*$

[OtherField]
Path=otherfield
NamePath=field_name
ValuePath=field_value/interesting
FieldSections=SomeField
AttributeSections=OtherFieldAttr,OtherFieldAttr2,OtherFieldAttr3

[OtherFieldAttr]
Path=^@(attribute)$

[OtherFieldAttr2]
NamePath=field_value/attr_name
ValuePath=field_value/attr_value

[OtherFieldAttr3]
Path=attrs/^.*$

To ingest the JSON, send the ingest action to CFS:

http://localhost:7000/action=ingest&adds=%3Cadds%3E%3Cadd%3E%3Csource%20
                                         filename%3D%22file.json%22%20
                                         lifetime%3D%22permanent%22%20%2F%3E
                                         %3C%2Fadd%3E%3C%2Fadds%3E

This would produce the following document:

#DREREFERENCE 12345abcd
#DREFIELD myfield="7"
#DREFIELD myfield/@attribute="true"
#DREFIELD myfield/@key="value"
#DREFIELD myfield/@attr1="fish"
#DREFIELD myfield/@attr2="chips"
#DREFIELD myfield/@attr3="peas"
#DREFIELD myfield/someField="12.9"
#DREFIELD someField="value1"
#DREFIELD someField="value2"
#DRESECTION 0
#DRECONTENT
page1
#DREENDDOC

#DREREFERENCE 12345abcd
#DRESECTION 1
#DRECONTENT
page2
#DREENDDOC

#DREREFERENCE 12345abcd
#DRESECTION 2
#DRECONTENT
page3
#DREENDDOC