File Action

Copy, move or delete the files an input, List Files or File from URL hands it through its Files anchor, before the data is processed or after the run succeeds or fails.

Category

Output

Connections

Amazon S3Azure Blob StorageGoogle Cloud StorageSFTPNetwork Share (SMB)

Also known as

copy files, move files, delete files, archive, transfer, ftp, share, unc, network drive, housekeeping, purge

What it does

Copies, moves or deletes files (the files themselves, not rows) on S3, Azure Blob, GCS, SFTP or a network share. It is just the action: its files come from the node wired to its Files anchor. Wired to an S3, Azure Blob, GCS, SFTP or Network Share input, it acts on the files that input lists for the run. Wired to List Files, it acts on every file listed, which is how to move files Trinium cannot read, such as PDFs and zips. Wired to File from URL, it copies the downloaded file. It runs first, before any row is processed, or after the run succeeds or fails. Within one connection the store copies the file itself; between two connections Trinium streams it across with no scratch disk, and a file is checked at its destination before anything removes it.

Inputs and outputs

When to use it

• Read files from a Windows network share and move them to S3 in the same run
• Archive the files an input read, but only once the whole run has succeeded
• Move a failed run's file to an error folder, so one bad file does not fail the run every night
• Clear out files older than 30 days: List Files with 'Only files older than', wired to a Delete
• Land a byte-for-byte copy of a downloaded file in S3

Examples

Network Share Files (folder exports, *.csv) · its Files anchor into File Action: Move · Runs first · to S3, folder raw/${date}

S3 Files · its Files anchor into File Action: Move · After the run succeeds · to archive/

S3 Files (Run each file separately) · its Files anchor into File Action: Move · After the run fails · to error/

List Files (folder landing, only files older than 30 days) · its Files anchor into File Action: Delete · Maximum files to delete: 5000

File from URL · its Files anchor into File Action: Copy · Runs first · to S3, folder raw/

Tips

• A File Action names no folder of its own. Wire the Files anchor of an input, List Files or File from URL into it; with nothing wired it shows a badge and cannot run.
• Runs first acts after the inputs have listed and downloaded their files, and before any row is processed, so a move there never changes what the run reads. If the run then fails, the files have already moved: to move only on success, use After the run succeeds, or Copy first and Move after success.
• A Test run acts on nothing. The run's messages say what it would do with each file.
• A copy is checked at its destination (its size) before a Move removes the source, and the source is removed only if it is still the file that was listed. A file that changed in the meantime is left where it is and named.
• Within one connection the store copies the file itself: S3 copies objects of any size (larger than 5 GB in parts), Azure is waited on until its copy is complete, and a network share copies on the server. Between two connections the bytes stream through Trinium with nothing written to its disk, and the run's messages record each file's SHA-256.
• A File Action runs as part of its pipeline's run, and a server runs one Staging or Production run at a time, so a long copy keeps other pipelines on that server waiting until it finishes. Put a large transfer in its own pipeline, or route it to a server of its own.
• An action that runs first fails the run if a file cannot be acted on, before any row is processed. One that runs after never changes the run's outcome: a file it could not act on is a warning that names it.
• If a file is already at the destination you choose what happens: stop, skip it, replace it, replace it only if the new one is newer, or keep both by adding a suffix to the new one. Stop is decided for every file before the first one moves.
• Delete acts only after the run succeeds, needs a Maximum files to delete, and only ever acts on files Trinium listed in this run. More files than the maximum stops the run before anything is touched.
• A file File from URL downloaded can only be copied: a link cannot be moved or deleted.
• When one node's Files anchor feeds several File Actions that act at the same time, the copies run first. Two that would both move or delete the same files are refused.
• Before you Move or Delete, ask whether another value of a parameter still needs the file. If each value has its own folder, set the input to landing/${region}/, Move After the run succeeds, and set the destination to archive/${region}/. If one folder holds every region and one run can read them all, drop the parameter, split the rows with a Route node, and Move After the run succeeds.
• If one folder holds every region and each region must run separately, leave File Action out of those runs and turn Bookmark on, so each value remembers what it read. Then build a Flow: Load EU and Load US, each going On success into an Archive pipeline, which is List Files on landing/ with Only files older than 1 day, wired into a File Action that Moves.

Related nodes