Partner and client file onboarding, with one rule for files that change shape.
Every partner sends a slightly different file, and one of them reordered the columns. Onboarding a new client's data eats integration time, and a column that quietly moved is how bad data gets in.
A script per partner, and a crash or a silent swap when one changes.
Each new partner means a new script, or a new branch in an old one. A file that arrives with a column added, missing or moved either crashes the job at 2am or, worse, loads values into the wrong fields and reports success.
The reject list goes back to the partner as a spreadsheet somebody builds by hand, and nobody can say which files were read last night and which were skipped.
What it replaces
- A script per partner → one input over the whole folder
- A crash, or values in the wrong fields → columns matched by name, every difference named
- A reject spreadsheet built by hand → the partner's own bad rows, attached to an email
- Polling on the hour → a run that starts when the file lands
- Which files did we read? → a list per environment that a run updates only when it succeeds
A folder in, valid rows loaded, rejects sent back to the partner.
The same pipeline reads CSV, JSON, XML, Parquet or Excel, over SFTP, S3, Azure Blob or Google Cloud Storage.
Built from these nodes, each with its own documentation page
- SFTP Files: a file or a whole folder, gzipped or zipped, combined into one input
- Data Validation: nine kinds of check, with failures on an output of their own
- Postgres output: upsert on a key, so a re-run corrects rather than duplicates
- Send Email: the partner's own rejected rows, attached as CSV or Excel
Four nodes and one setting that decides what a different file means.
The drift rule is the same for all five formats and all four stores, so a partner who switches from CSV to Excel does not change how the pipeline behaves.
Point SFTP Files at the drop folder.
Pick the partner's connection, a folder and a name pattern. A .csv.gz is read automatically, and a .zip is opened when the pattern names it. With Bookmark on, a run reads only the files no successful run has read, each remembered by name, size and modified time. To archive each file once a run succeeds, wire the input's Files anchor to a File Action set to Move.
folder: incoming/acme pattern: *.csv Bookmark: on
File Action: Move · after the run succeeds · to archive/acme/
Wait for a file to finish arriving.
On a Staging or Production run, a file is read only once its size and modified time have stayed the same for 30 seconds, so a file still uploading is not read half-written. One still changing after five minutes is left in place for the next run and named as a warning.
Decide what a different file means.
If a file is different: Reconcile, the default, reads any file that shares at least one column name with the first, by name, with missing columns filled with nulls and extra columns dropped. A file that shares no columns is skipped. Skip reads only files that match exactly, and Stop the run fails on any difference.
If a file is different: Reconcile
Isolate each file if you need to.
Turn on Run each file separately and every file becomes its own run, so one bad file never fails the rest of the night's files.
Gate the rows with Data Validation.
Required values, ranges, lengths, patterns, allowed values and rules across columns. Give a rule a message in business terms and that is what the partner reads in the errors column.
account_id: Not null · currency: Allowed values USD, EUR, GBP · Message: "Currency must be USD, EUR or GBP"
Load the good rows, and send the rest back.
Postgres output upserts on the account key. The Invalid output goes to Send Email, one message per run to the partner, with their rejected rows attached. An Allowed recipient domains list on the email connection means it can only ever deliver to the domains you name.
Postgres output: upsert on account_id · Send Email: one email per run, attach as CSV
The run starts when the partner's file lands.
A watch on the folder
A watch schedule fires only when a new file has stopped changing between two checks, so no run waits on a file that is still crawling in over a slow link.
A watch that cannot look says so
If the folder cannot be listed three times in a row, the schedule records a failed run naming the error, so a partner who changed their password does not go unnoticed. One blip alerts nobody.
Staging never steals from Production
Staging and Production each keep their own list of processed files, and a Test run reads a sample and moves, deletes or bookmarks nothing.
A success can still have something to say
A run that skipped a file or quarantined rows succeeds with a warning count, and the schedule's alert can fire when a run finishes with warnings.
A partner who reordered the columns is a line you can read.
Every difference between a file and the first one is named in the run's Messages, grouped per file, so a missing column is said once rather than once a row. These are the product's own words for four files that changed shape in four different ways.
A file Trinium could not read, such as a corrupt zip, is skipped by name or stops the run, whichever you chose, and a File Action moving the files the run read leaves it where it is.
acme_2026-10-02.csv: columns are in a different order to 'acme_2026-10-01.csv' — matched by name acme_2026-10-03.csv: column 'region' missing — filled with nulls acme_2026-10-04.csv: extra column 'promo_code' — dropped (the columns come from 'acme_2026-10-01.csv') photos.csv: shares no columns with 'acme_2026-10-01.csv' — it has 'file', 'width', 'height' — file skipped
Scope worth knowing. The first file read sets the columns, so an unrelated file that sorts first becomes the contract: a name pattern keeps strangers out. Fixed-width text and EDI documents (X12, EDIFACT, HL7) are not read.
Read further
Other jobs Trinium takes over
One graph instead of a function, a state machine and a crawler
The nightly warehouse loadForty tables, one schedule, one place to look when it breaks
Reconciliation you can hand overMonth-end close, and proving a migration landed
Bring the partner whose files cause the most trouble.
Thirty days of the full Enterprise product on your own server, against your own data. If you would rather see it first, book a walkthrough with the people who build it.