Bookmarks
A bookmark remembers what one input has already loaded, so the next Live run reads only what is new.
Two kinds
- A file input (S3, Azure Blob, Google Cloud Storage, SFTP and Network Share) with Bookmark on remembers each file it read, by name and version. A file dropped again with new contents counts as new.
- A database input with a Bookmark column remembers the largest value of that column it read. The next run reads only the rows after it. A Lookback re-reads the N minutes before that point, to catch rows a long transaction committed late; a first run and every Test run read exactly from Start from.
When a bookmark moves
Only a Live run in Staging or Production that succeeds moves a bookmark. A run that fails or is cancelled moves nothing, so the next run reads everything since the last success. A Test run reads from the start and never moves a bookmark. Two Live runs that share a bookmark never overlap, so they never read the same files or rows at once. Runs with different values of a text parameter each keep their own and may overlap: see When runs overlap.
What a bookmark is kept for
Each input keeps one bookmark per environment, and per value of each text parameter the pipeline uses. Staging's runs never move Production's bookmark. When ${region} appears anywhere in the pipeline, in the input's folder or query, in a Filter after it, or in an output's name, region = EU and region = US each read only their own new files and rows. A value used only as a label, such as in an email subject, splits the bookmark all the same, and a value with no bookmark yet reads from the start.
Number, true/false and date parameters do not split a bookmark. A number or true/false parameter in a bookmarked table's name is refused by name, because one bookmark would serve every value of it. Make it a text parameter with a list of choices instead.
What never changes a bookmark
- Dates.
${date}in a folder or a table name moves to a new day without touching the bookmark. A database input readingevents_${date:%Y%m%d}carries on into each new day's table after yesterday's value, so bookmark a date-time column there. - Editing the input. Changing a database input's table, query, connection or column keeps its bookmark. When a run reads something other than what its bookmark last read, its Messages say so. A Bookmark column of another kind, a date where there was a number, is refused until you Set or Reset it.
Changing which text parameters a pipeline uses
- Start using one, and each value carries on from the bookmark kept before. Nothing is read twice.
- Stop using one, or rename it, and runs stop with a message naming the node, because only you can say whether a file that EU read holds US rows nobody loaded yet. For a database input, Set where each value starts, or Reset. For a file input, press Reset processed files… and choose Carry on with new files only, where every file any value read counts as read, or Read every file again.
- After a rename, each value of the new name needs its own Set.
Set and Reset
Both are in the input's settings, and both wait while a Live run of that pipeline and environment is going, whichever value it runs with.
- Set… chooses where one value's next run starts on a database input. Type the value to read after; to read everything again, type a value just before Start from. Each value shows when its bookmark last moved, and which run moved it or who set it.
- Reset forgets the bookmark of every value in one environment at once. A database input then reads from Start from, and a file input reads every file again.
A change in Production needs the pipeline's name typed, and Production access on the input's connection. Every change is recorded on the pipeline's Timeline.
${run_id} names no folder
A run's id is new on every run, so an input whose folder, file name or pattern holds ${run_id} could never find a file. Every start refuses it by name, and the builder marks the node. Use ${run_id} where a run writes: an output's path or a File Action's destination.