The nightly warehouse load, orchestrated and observable.
Forty tables have to be in the warehouse by 6am, and somebody has to know if they aren't. The load itself is not hard. What is hard is the part around it.
Knowing which of the forty failed, why, and who found out.
Each job fires at a time somebody guessed, so a slow upstream means the next job reads yesterday's data and reports success. A failure stops one job and lets every job after it run on whatever was there.
The first person to know is whoever opens the dashboard at 9am, and the logs are in as many places as there are jobs.
What it replaces
- A cron entry per job, with guessed gaps → wires that wait for the step before
- A try/except someone added later → an on-failure branch drawn on the canvas
- Which one failed? → a flow run with every step's status
- A hand-edited backfill → a date handed to each step as a parameter
- A separate orchestrator → included in both editions
Each load is a pipeline. The night is a Flow of pipelines.
A pipeline is a graph of rows. A Flow is a graph of whole pipeline runs: the layer that decides what runs after what, and what happens when something fails.
Built from these nodes, each with its own documentation page
- Postgres input: a table or a query, with a bookmark so each run reads only new rows
- Data Cleanse: the tidy-up between a system you do not control and a warehouse that is strict
- Snowflake output: one of nine warehouse and database outputs, with pre-SQL and post-SQL around the load
- Postgres output: append, truncate or upsert, in one transaction by default
One pipeline per source, then one Flow for the night.
Warehouse and database nodes cover Snowflake, BigQuery, Databricks, Amazon Redshift, ClickHouse, Trino, PostgreSQL, SQL Server, MySQL and MongoDB. Each reads its columns without a run.
Read only what changed.
Give the source a database bookmark on an updated-at or id column and each run reads only the rows past where the last successful run stopped. The bookmark moves only when the whole run succeeds, so a failed night is read again in full. Bookmarks work on Postgres, MySQL, SQL Server, Redshift, ClickHouse and Trino sources.
Bookmark column: updated_at · Start from: 2026-01-01 00:00:00
Load so a re-run is safe.
Use upsert on a key where the output offers it, or the standard warehouse pattern: empty a staging table in pre-SQL, load it, and MERGE in post-SQL. Post-SQL runs only after the load has finished successfully, so a failed load never fires the merge.
Pre-SQL: TRUNCATE TABLE STG_ORDERS Post-SQL: MERGE INTO ORDERS t USING STG_ORDERS s ON t.ID = s.ID …
Decide what a failed load leaves behind.
Postgres and Redshift outputs load in one transaction by default. SQL Server and MySQL outputs, a Trino append and a MongoDB replace do the same when you turn on All or nothing, so a load that fails part way leaves the table as it was.
Draw the night as a Flow.
Each step runs a whole pipeline. Wire two steps and the second waits for the first; a step with several incoming wires waits for all of them. A wire carries on success, on failure or always, and when a step fails, everything that depends on it is skipped instead of running on bad data.
Hand each step its date.
A Flow can carry parameters and hand them to its steps, so one flow run reads one day everywhere. A date value can be ${date-1}, read from the moment the flow run was queued.
Run pipeline · orders_to_warehouse · run_date = ${date-1}
Tell the right people.
A notify step emails a person or posts to Slack or Discord, naming the step that failed. The schedule's own alert can fire on failure, on success, or when a run succeeds with warnings.
Scheduled in Production, watched from one screen.
A cron schedule in Production
The Flow's schedule fires in your instance's timezone and runs each step at the version deployed to Production. A fire is skipped while the previous run of that schedule is still going, so two nights never interleave.
A time limit on every run
A run that passes its time limit, 12 hours by default or one you set per pipeline, is stopped and recorded as failed, so the alert fires instead of the run hanging until morning.
Side by side, when you have the servers
Steps with no wire between them never wait for each other. Each server runs one production pipeline at a time and uses all of it, so a second server runs a second load at the same moment.
Every run answers what happened
Each node records its rows in and out, its duration and the time it spent working, and each run records its peak memory and how long it queued.
At 6:04am, one screen names the step and the node.
The flow run shows every step's status, and a failed step carries its pipeline's own error, with the node named as it is on your canvas rather than as an id. The alert lands where your team already looks.
If a server dies mid-run, the run is failed by name within minutes and reported. It is never silently re-run against your warehouse, because re-running a half-finished load is how rows get duplicated.
Flow "nightly-close" — step "Alert the data team" fired. Triggered by: "Reconcile" (Failed)
Scope worth knowing. A Flow does not loop and has no wait step, and a later step cannot read a value an earlier step produced: pipelines hand data to each other through a table or a folder. Snowflake, Databricks and BigQuery loads are not all-or-nothing; a MERGE from a staging table is how they become safe to repeat.
Bring the night that pages you most.
Thirty days of the full Enterprise product on your own server, against your own data. If you would rather see it first, book a walkthrough with the people who build it.