Incremental Synchronize Example

This section describes the connector in operation.

Cycle 1

To begin with, the datastore table is empty. The “data” directory (the repository from which the sample connector retrieves data) contains three files:

       File1.txt
       File2.txt
       File3.txt

When the synchronize action runs, the connector loops over all the files. For each file, it adds a record to the table with a pending action of “Add”. When this is finished the datastore table looks like this:

Reference LastModified PendingAction Seen
File1.txt 2013-09-05 12:00 Add SEEN
File2.txt 2013-09-05 12:05 Add SEEN
File3.txt 2013-09-05 12:10 Add SEEN

The connector then updates all UNSEEN records so that their PendingAction is "Remove", but in this case no records are UNSEEN.

Next, the connector performs the actions in the table, sending an ingest add for each file. File1.txt and File2.txt are ingested successfully and the ingest result handler updates the PendingAction to “None” for those records. Normally File3.txt would be ingested successfully, but in this case it is not. This could be due to a network error, or the connector being stopped before the cycle is complete. As the result handler is not called (or is called but with its success parameter set to false), the PendingAction is not changed.

After the end of the first cycle the datastore looks like this:

Reference LastModified PendingAction Seen
File1.txt 2013-09-05 12:00 None SEEN
File2.txt 2013-09-05 12:05 None SEEN
File3.txt 2013-09-05 12:10 Add SEEN

Cycle 2

At the next synchronize cycle the data directory has been modified. File1.txt has been deleted, and File2.txt has been modified. File3.txt has not changed.

When the synchronize action runs, the connector marks all records as "UNSEEN":

Reference LastModified PendingAction Seen
File1.txt 2013-09-05 12:00 None UNSEEN
File2.txt 2013-09-05 12:05 None UNSEEN
File3.txt 2013-09-05 12:10 Add UNSEEN

The connector then loops over all files in the repository, updating the PendingAction column. File2.txt has a different modified date so its PendingAction is set to "Replace". File3.txt is unchanged so the record is not modified. After this the datastore table looks like this:

Reference LastModified PendingAction Seen
File1.txt 2013-09-05 12:00 None UNSEEN
File2.txt 2013-09-05 12:30 Replace SEEN
File3.txt 2013-09-05 12:10 Add SEEN

Next, the connector sets the PendingAction for all "UNSEEN" records to "Remove":

Reference LastModified PendingAction Seen
File1.txt 2013-09-05 12:00 Remove UNSEEN
File2.txt 2013-09-05 12:30 Replace SEEN
File3.txt 2013-09-05 12:10 Add SEEN

Finally, the connector performs each of the pending actions. This time the connector is allowed to finish, there are no network errors and all of the ingest commands are successful. The ingest result handler removes the record for File1.txt, and sets the PendingAction of the other records to "None":

Reference LastModified PendingAction Seen
File2.txt 2013-09-05 12:30 None SEEN
File3.txt 2013-09-05 12:10 None SEEN