Retire the scripts and cron jobs only one person understands.
The pipeline works. The problem is that one person understands it: a folder of Python, a crontab, and a README that stopped being true two years ago.
It runs fine, until its author is on holiday and it doesn't.
Nobody else can tell where the data comes from, what happens to it or where it lands without reading the code. A failure is a stack trace in a log file on one server, if anyone looks.
Changing it means editing a script in production, because there is no version to roll back to and no staging copy to try it on.
What it replaces
- The crontab → a schedule you can see, edit and pause
- git log, maybe → numbered versions you can compare, deploy or restore
- print() and a log file → rows, timings and errors for every node of every run
- “Did it run?” → an alert to a person or a channel when it does not
- A .env nobody rotates → connections encrypted at rest, never in the pipeline
On a canvas, the graph is the documentation.
Anyone can open it and see where the data comes from, what happens to it and where it lands. The transformations are still real work, written in ordinary SQL.
Built from these nodes, each with its own documentation page
- REST API (Read): pages through a JSON API with the credential held on the connection
- Formula: new and rewritten columns in SQL, checked as you type
- Filter: a condition, with the rows that fail it on their own output
- Postgres output: append, truncate or upsert, in one transaction by default
- API Lookup: the part that really is bespoke, kept as a service you run and called for each row
Rebuild the script node by node, and call the bespoke part where it runs.
Most of what a data script does is read, reshape, filter and load. Those become nodes. What is left runs where you already run code, and the pipeline calls it.
Read the API with REST API (Read).
The connection holds the base URL and the credential; the node walks the pages for you. Fetch a sample, declare the columns you want, and every node downstream sees them before a run.
Move the transformations into Formula.
Each output column is a SQL expression, validated against the real inbound columns as you type, so a misspelled column is caught in the builder rather than at 3am. Earlier rows can build on later ones in readable steps.
total = [price] * [quantity] band = CASE WHEN [total] >= 1000 THEN 'large' ELSE 'small' END
Keep the rows you want with Filter.
One condition; matches go to True and the rest to False, so the rows you dropped are still there to look at.
[status] = 'active' AND [total] > 0
Load it so a re-run is safe.
Postgres output upserts on a key, so running the job twice corrects rather than duplicates, and the load is one transaction by default.
Mode: upsert · Key columns: id
Call the truly bespoke part as a service.
Put the piece no node covers behind a small HTTPS endpoint on your own infrastructure: a function, a container or an internal service. API Lookup sends it each row, with the row's values in the path, the query or the body, and adds its answer as new columns.
GET /enrich/${id}?total=${total}
The same schedule, with a version, a record and an alert.
A schedule instead of a crontab
Fires in your instance's timezone, runs the version deployed to Production, and is paused in its own dialog, where you can see what you are pausing.
Staging before Production
Every connection has a staging side and a production side, so the same pipeline reads your staging database in Staging and your production database in Production.
A rollback is one deploy
Every version is numbered and never changes. Deploy a version Production ran before, with a note saying why.
Alerts that name the node
A failure emails a person or posts to a channel, and names the node as it is on your canvas.
A failure names the node, the step and the reason.
Every node records its status, rows in and out, duration and error as the run goes, and the canvas shows where it stopped. If the server running it dies, the run is failed by name within minutes and never silently re-run against your targets.
When a call to your own service fails, API Lookup either stops the run or keeps the row with its HTTP status, as you choose, so a bad answer is a row you can look at rather than a log line.
# one request per row, the row's values in the URL
GET /enrich/${id}?total=${total}
add status http_status
on failure keep the row and carry on
Scope worth knowing. A script that runs shell commands does not carry over, because the server runs none. Code that needs a library no node offers stays a service of yours, and API Lookup calls it.
Other jobs Trinium takes over
One graph instead of a function, a state machine and a crawler
The nightly warehouse loadForty tables, one schedule, one place to look when it breaks
Sending results back to the businessThe answer, delivered where people already work
Bring the script nobody else dares to touch.
Thirty days of the full Enterprise product on your own server, against your own data. If you would rather see it first, book a walkthrough with the people who build it.