S3 Files

Read a single object or a whole prefix from an S3 bucket. A Bookmark can read only the files no successful run has read.

Category

Input

Connection

Amazon S3

What it does

Reads files from an S3 bucket through a saved Connection: a single object, or every object under a prefix (with an optional name pattern) combined into one run. Decodes with one chosen format (CSV, JSON, Parquet, Excel, XML). With Bookmark on, it reads only files no successful run has read. To move, copy or delete the files after the run, wire its Files anchor to a File Action.

Inputs and outputs

When to use it

• Ingest a raw/landing bucket on a schedule
• Process only new files each run without touching the bucket (Bookmark)
• Archive the files once they're processed: wire the Files anchor to a File Action set to Move, after the run succeeds

Examples

prefix: raw/sales/   pattern: *.csv

prefix: raw/sales/dt=${date-1}/

key: raw/sales/2026-01.csv

Bookmark: on

prefix: inbound/   pattern: *.zip

Tips

• The format settings are the same as on the CSV Files, JSON Files, Excel Files and XML Files inputs: the delimiter, the encoding and the rows to skip, which sheet to read, and the record element.
• Turn on “Add source file name as column” to keep each row's object key (exports/orders/dt=2026-09-01/orders.csv), so you can tell which folder it came from. The run's Messages name a file by its full key too.
• Tokens filled in at run time, in the prefix and the object key: ${date} (today in the instance timezone, 2026-08-26), ${date-1} / ${date+7} (a day offset), ${date:%Y%m%d} (a layout) and ${datetime} (UTC, filename-safe). A ${run_id} is refused here: a run's id is new on every run, so it names a run's own output, never a folder to read. A misspelled token fails the run by name before anything is downloaded; a pipeline parameter (${region}) works in the same path.
• To mix formats (e.g. CSV and Excel) in one flow, use two S3 nodes into a Union.
• Bookmark records the files only after a full, successful Staging or Production run, never a Test run, so a run that fails reads them again next time.
• Bookmark keeps the bucket immutable. Staging and Production each keep their own list of processed files, and so does each value of a text parameter the pipeline uses, so testing in Staging never makes Production skip a file and region = US never skips a file region = EU read. Reset a list to reprocess everything; after a text parameter is removed or renamed, runs stop until Reset, which also offers to carry on with new files only.
• Turn on 'Run each file separately' to process every object as its own run, named by its full key. Every start does this: Run Now, a schedule, a file watch, a flow step and the Run API. With Bookmark on, only the objects no successful run has read get a run, up to 500 a start; the rest are read by the next start. When there is nothing new to read, the start makes one run that reads nothing. The runs take turns, one at a time.
• A lake laid out in key=value folders (table/reading_date=20260901/part-0.parquet)? Turn on 'Read folder names as columns' and the folder becomes a text column on every row; add 'Only read folders where' ([reading_date] >= to_char(now() - INTERVAL '62 days', '%Y%m%d')) and only those folders are downloaded at all.
• Compressed files: a .csv, .tsv, .json, .ndjson or .xml file ending in .gz, .bz2 or .xz is read automatically. An archive (.zip, .tar, .tar.gz or .tgz, .tar.bz2, .tar.xz) is opened when Name pattern names it (*.zip, *.tgz) or the key points at one: every file of the chosen format inside is read (Excel and Parquet included) and the rest are ignored; the run's Messages name a file inside one by the archive's key and the file's name, as exports/2026/export.zip!data.csv. Inside a tar only ordinary files are read: a link, a device or a path that leads out of the archive is named in Messages and skipped. A file named only .gz needs a pattern; Parquet and Excel are read compressed only inside an archive.
• A file, a compressed file or an archive that cannot be read (truncated, corrupt, password-protected, not really of its format) follows 'If a file is different', wherever it sorts: skipped and named in the run's Messages (and in the run's warning count), or the run stops under 'Stop the run'. The columns come from the first file that can be read, in the builder as in the run. A File Action moving the files the input read leaves it where it is.

Related nodes