SFTP Files

Read a single file or a whole folder from an SFTP server. A Bookmark can read only the files no successful run has read.

Category

Input

Connection

SFTP

What it does

Reads files from an SFTP server through a saved Connection: a single file, or every file in a folder (with an optional name pattern) combined into one run. Decodes with one chosen format (CSV, JSON, Parquet, Excel, XML). With Bookmark on, it reads only files no successful run has read. To move, copy or delete the files after the run, wire its Files anchor to a File Action.

Inputs and outputs

When to use it

• Pick up files a partner, bank, or vendor drops on a schedule
• Process only the new drops each run and leave everything else alone (Bookmark)
• Archive the files once they're loaded: wire the Files anchor to a File Action set to Move, after the run succeeds

Examples

folder: incoming/sales   pattern: *.csv

folder: incoming/sales/${date-1}

path: incoming/sales/2026-01.csv

Bookmark: on

folder: inbound/   pattern: *.zip

Tips

• The format settings are the same as on the CSV Files, JSON Files, Excel Files and XML Files inputs: the delimiter, the encoding and the rows to skip, which sheet to read, and the record element.
• Turn on “Add source file name as column” to keep each row's file path (inbound/acme/2026-09-01/orders.csv), so you can tell which folder it came from. The run's Messages name a file by its full path too.
• Tokens filled in at run time, in the prefix and the object key: ${date} (today in the instance timezone, 2026-08-26), ${date-1} / ${date+7} (a day offset), ${date:%Y%m%d} (a layout) and ${datetime} (UTC, filename-safe). A ${run_id} is refused here: a run's id is new on every run, so it names a run's own output, never a folder to read. A misspelled token fails the run by name before anything is downloaded; a pipeline parameter (${region}) works in the same path.
• Paths are relative to the Connection's base folder, so moving the account's root doesn't break the pipeline.
• To mix formats (e.g. CSV and Excel) in one flow, use two SFTP nodes into a Union.
• Bookmark records the files only after a full, successful Staging or Production run, never a Test run, so a run that fails reads them again next time.
• A Staging or Production run reads a file only once its size and modified time have stayed the same for 30 seconds, so a file a partner is still uploading is not read half-written. It waits up to 'Wait for files still arriving' (5 minutes unless set; 0 turns it off); a file still changing then is left in place for the next run and named as a warning. A file watch fires only once a new file looks the same at two checks. The check tells a file still arriving from one that has stopped. It cannot tell a finished file from an upload that died half way, nor from a link so slow that a single write takes longer than 30 seconds to arrive (below about 8 KB/s for a recent OpenSSH client, which writes 255 KB at a time). Only a partner that uploads under a temporary name, or writes a marker file, proves a file complete.
• Bookmark leaves the server untouched. Staging and Production each keep their own list of processed files, and so does each value of a text parameter the pipeline uses, so testing in Staging never makes Production skip a file and region = US never skips a file region = EU read. Reset a list to reprocess everything; after a text parameter is removed or renamed, runs stop until Reset, which also offers to carry on with new files only.
• Turn on 'Run each file separately' to process every file as its own run, handy when one bad file shouldn't fail the rest, named by its full path. Every start does this: Run Now, a schedule, a file watch, a flow step and the Run API. With Bookmark on, only the files no successful run has read get a run, up to 500 a start; the rest are read by the next start. When there is nothing new to read, the start makes one run that reads nothing. The runs take turns, one at a time.
• A lake laid out in key=value folders (table/reading_date=20260901/part-0.parquet)? Turn on 'Read folder names as columns' and the folder becomes a text column on every row; add 'Only read folders where' ([reading_date] >= to_char(now() - INTERVAL '62 days', '%Y%m%d')) and only those folders are downloaded at all.
• A schedule can watch this folder and fire only when something new lands, instead of polling on a clock.
• Compressed files: a .csv, .tsv, .json, .ndjson or .xml file ending in .gz, .bz2 or .xz is read automatically. An archive (.zip, .tar, .tar.gz or .tgz, .tar.bz2, .tar.xz) is opened when Name pattern names it (*.zip, *.tgz) or the key points at one: every file of the chosen format inside is read (Excel and Parquet included) and the rest are ignored; the run's Messages name a file inside one by the archive's key and the file's name, as exports/2026/export.zip!data.csv. Inside a tar only ordinary files are read: a link, a device or a path that leads out of the archive is named in Messages and skipped. A file named only .gz needs a pattern; Parquet and Excel are read compressed only inside an archive.
• A file, a compressed file or an archive that cannot be read (truncated, corrupt, password-protected, not really of its format) follows 'If a file is different', wherever it sorts: skipped and named in the run's Messages (and in the run's warning count), or the run stops under 'Stop the run'. The columns come from the first file that can be read, in the builder as in the run. A File Action moving the files the input read leaves it where it is.

Related nodes