Data-lake pipelines, without the cloud glue.
A function to fetch it, a state machine to sequence it, a crawler to notice the schema, a job to transform it. In Trinium the same job is one pipeline you can see, test on real rows and roll back.
One job, spread across four services, with four consoles and four ways to fail.
Nobody designed it that way. It accreted: a function was the quickest way to fetch the files, a state machine was the quickest way to order the steps, and a crawler was the quickest way to tell the query engine what the columns are.
Each piece has its own console, its own logs and its own way to break. When the 2am load comes up short, somebody opens four tabs to find out which piece did it, and the schema the crawler recorded last week is still the one the transform believes.
What it replaces
- A function per step → nodes on one canvas, all running at once
- A state machine → a Flow, with the same sequencing and branching
- A crawler on a schedule → columns read at design time, with no run
- Date-partitioned folders → read as a column, written one folder per value
- Four consoles → one run screen, per node, with the logs beside it
- Several meters → one price per server, the same every month
Raw files in, validated Parquet or Iceberg out, rejects kept aside.
The same shape works on all four cloud stores: Amazon S3, Azure Blob Storage, Google Cloud Storage and SFTP. Swap the input and output nodes and nothing in between changes.
Built from these nodes, each with its own documentation page
- S3 Files: a whole prefix of CSV, JSON, Parquet, Excel or XML files, gzipped or zipped, combined into one input
- Data Validation: rules on every row, with failures leaving on an output of their own
- Data Cleanse: trimming, blanks to nulls and standard case before anything is loaded
- S3 File: Parquet, CSV, JSON or Excel, one folder per partition value if you ask
- Apache Iceberg output: on Enterprise, a commit to an Iceberg table through AWS Glue, Amazon S3 Tables or a REST catalog
Five nodes, and each one is a form, not a script.
Every field that takes a column offers the real columns, typed, the moment the node is wired. Nothing below needs a run to configure.
Point S3 Files at the raw zone.
Pick a saved S3 connection, a prefix and a name pattern. Turn on Bookmark and each run reads only the objects no successful run has read, while the bucket stays untouched. The columns appear at once, read from the CSV header, the Parquet footer or the catalog.
prefix: raw/orders/ pattern: *.csv Bookmark: on
Read the folders as columns, and skip the ones you do not need.
If the lake is laid out as reading_date=2026-09-01/ folders, turn on Read folder names as columns and the folder becomes a column on every row. Add a condition and only the folders it keeps are downloaded at all.
Only read folders where: [reading_date] >= to_char(now() - INTERVAL '62 days', '%Y%m%d')
Gate the rows with Data Validation.
Declare what a good row looks like: required values, ranges, patterns, a fixed vocabulary, or a rule across columns. Rows that pass leave on Valid; rows that break a rule leave on Invalid with a _validation_errors column naming every rule they broke. A structure rule, such as a column that must exist, fails the run instead, because a changed source is not a bad row.
order_id: Not null · amount: Range 0 to 1000000 · status: Allowed values open, shipped, returned
Tidy what passed, and keep what failed.
Data Cleanse trims whitespace and turns blanks into nulls. The Invalid output goes to its own S3 File, so a rejected row is kept with its reason rather than lost or loaded.
quarantine/orders/${date}/rejects.csv
Write the curated zone.
S3 File writes Parquet. Partition by a column and rows are routed into column=value/ folders, the layout every query engine reads. Set the write mode to Replace the folder and a run replaces only the folders it writes, so re-running a late day replaces that day and nothing else. On Enterprise, write an Iceberg table through AWS Glue instead: the write is the catalog commit, so Athena reads it the moment the run finishes.
destination: curated/orders/ partition by: order_date if it exists: Replace the folder
It starts when the files land, and production only runs what you deployed.
When new files arrive
A watch schedule looks at the prefix and starts a run within about a minute of new objects landing, instead of polling on a guessed time. Staging and Production keep separate lists of what they have read, so testing in Staging never makes Production skip a file.
On a clock, in your timezone
Paths take ${date} and ${date-1}, read from one clock per run in your instance's timezone, so a nightly job reads yesterday's folder without anyone editing it. A misspelled token fails the run by name before anything is downloaded.
Test on 50 rows first
Press Test and the draft runs against a 50-row sample, previews every port, and writes nothing to the bucket. Then deploy a version to Staging, which reads the staging side of each connection, and the same version on to Production.
In order with everything else
A Flow runs this pipeline, then the ones that depend on it, skips what depends on a failure, and tells someone by email, Slack or Discord.
A short load says which file and which rule, in words.
Every row that failed validation is in the quarantine file with the rules it broke, and the run's Messages say how many. A file that cannot be read, such as a truncated .gz, is skipped by name or stops the run, whichever you chose, and a File Action acting on the files the run read leaves it where it is.
If the server running it dies, the run is failed by name within minutes and reported. It is never re-run against your bucket on its own, because re-running half a load is how rows get duplicated.
| order_id | amount | status | _validation_errors |
|---|---|---|---|
| 120.00 | shipped | Not null [order_id] | |
| 10457 | -35.00 | open | Range [amount] |
| 10462 | 88.10 | pending | Allowed values [status] |
Scope worth knowing. Plain Parquet files are not registered in any catalog, so a query engine reads them through a table you define once. Write the curated zone as Iceberg through AWS Glue instead and every run's commit lands in Glue, with nothing else to set up.
Read further
Rebuild your lake's busiest job during the trial.
Thirty days of the full Enterprise product on your own server, against your own data. If you would rather see it first, book a walkthrough with the people who build it.