Incremental Synchronize Example
This section describes the connector in operation.
Cycle 1
To begin with, the datastore table is empty. The “data” directory (the repository from which the sample connector retrieves data) contains three files:
File1.txt File2.txt File3.txt
When the synchronize
action runs, the connector loops over all the files. For each file, it adds a record to the table with a pending action of “Add”. When this is finished the datastore table looks like this:
Reference | LastModified | PendingAction | Seen |
---|---|---|---|
File1.txt | 2013-09-05 12:00 | Add | SEEN |
File2.txt | 2013-09-05 12:05 | Add | SEEN |
File3.txt | 2013-09-05 12:10 | Add | SEEN |
The connector then updates all UNSEEN records so that their PendingAction is "Remove", but in this case no records are UNSEEN.
Next, the connector performs the actions in the table, sending an ingest add for each file. File1.txt
and File2.txt
are ingested successfully and the ingest result handler updates the PendingAction to “None” for those records. Normally File3.txt
would be ingested successfully, but in this case it is not. This could be due to a network error, or the connector being stopped before the cycle is complete. As the result handler is not called (or is called but with its success
parameter set to false
), the PendingAction is not changed.
After the end of the first cycle the datastore looks like this:
Reference | LastModified | PendingAction | Seen |
---|---|---|---|
File1.txt | 2013-09-05 12:00 | None | SEEN |
File2.txt | 2013-09-05 12:05 | None | SEEN |
File3.txt | 2013-09-05 12:10 | Add | SEEN |
Cycle 2
At the next synchronize cycle the data directory has been modified. File1.txt
has been deleted, and File2.txt
has been modified. File3.txt
has not changed.
When the synchronize
action runs, the connector marks all records as "UNSEEN":
Reference | LastModified | PendingAction | Seen |
---|---|---|---|
File1.txt | 2013-09-05 12:00 | None | UNSEEN |
File2.txt | 2013-09-05 12:05 | None | UNSEEN |
File3.txt | 2013-09-05 12:10 | Add | UNSEEN |
The connector then loops over all files in the repository, updating the PendingAction column. File2.txt
has a different modified date so its PendingAction is set to "Replace". File3.txt
is unchanged so the record is not modified. After this the datastore table looks like this:
Reference | LastModified | PendingAction | Seen |
---|---|---|---|
File1.txt | 2013-09-05 12:00 | None | UNSEEN |
File2.txt | 2013-09-05 12:30 | Replace | SEEN |
File3.txt | 2013-09-05 12:10 | Add | SEEN |
Next, the connector sets the PendingAction for all "UNSEEN" records to "Remove":
Reference | LastModified | PendingAction | Seen |
---|---|---|---|
File1.txt | 2013-09-05 12:00 | Remove | UNSEEN |
File2.txt | 2013-09-05 12:30 | Replace | SEEN |
File3.txt | 2013-09-05 12:10 | Add | SEEN |
Finally, the connector performs each of the pending actions. This time the connector is allowed to finish, there are no network errors and all of the ingest commands are successful. The ingest result handler removes the record for File1.txt
, and sets the PendingAction of the other records to "None":
Reference | LastModified | PendingAction | Seen |
---|---|---|---|
File2.txt | 2013-09-05 12:30 | None | SEEN |
File3.txt | 2013-09-05 12:10 | None | SEEN |