List Files

List the files in a folder as rows, with each one's path, name, size and modified time, without reading any of them.

Category

Input

Connections

Amazon S3Azure Blob StorageGoogle Cloud StorageSFTPNetwork Share (SMB)

Also known as

directory, folder, listing, inventory, metadata, audit, clean up, purge, older than

What it does

Lists the files in a folder as rows: one row per file, with its path, name, extension, parent folder, size, last-modified time and a change token. It never opens a file, so it works for PDFs, images, archives and anything else Trinium cannot parse; the file name is the only data it can get from those, and this is how it gets it. Point it at a saved S3, Azure Blob, GCS, SFTP or Network Share connection: the store is worked out from the connection you pick.

Inputs and outputs

When to use it

• Reconcile a document store against a system of record: match invoice numbers in file names against your ERP and report both sides that don't line up
• Find the gap in a partitioned lake: list every partition and compare it against the dates you expect, without reading the data
• Report on an archive that only grows: how much is there, how old is it, and is anything zero bytes from a failed write
• Route on a file name or size before deciding what to load
• Clean up a folder: list the files older than 30 days and wire its Files anchor to a File Action that deletes them
• Move or copy files Trinium cannot read (PDFs, images, archives) by wiring its Files anchor to a File Action

Examples

folder: raw/daily/   name pattern: *.csv

folder: lake/events/   include sub-folders: on

name pattern: *.pdf

Tips

• The folder accepts the same tokens as an input's folder: ${date}, ${date-1} / ${date+7}, ${date:%Y%m%d} and ${datetime}. A ${run_id} is refused here, because a run's id is new on every run and names no folder that exists yet. The day is today in the instance timezone, so a scheduled inventory can list yesterday's drop folder.
• On its own it never moves, deletes or bookmarks anything, and it never marks a file as processed, so a reader pointed at the same folder is completely unaffected.
• Its Files anchor hands every file it listed to a File Action, which copies, moves or deletes them.
• 'Only files older than' lists only files last modified more than that many days ago. A file whose modified time the store does not report is never counted as old. Maximum files still counts every file in the folder.
• Maximum files is what you EXPECT to find. If the folder holds more, the run stops and says so rather than reporting a partial listing as if it were the whole answer.
• The server has its own ceiling as well, sized from its memory, which you cannot raise from here. Narrow the folder or add a name pattern if you hit it.
• A Test run lists only the first 50 files and says so.
• change_token is the store's ETag (or, on SFTP, the modified time and size). When it changes, the file has changed.
• To then READ the files you found, use the S3, Azure Blob, GCS, SFTP or Network Share file nodes beside this one; with Bookmark on they read only what is new since last time.

Related nodes