Moving to Trinium
Moving from AWS Glue and Step Functions
A Glue job becomes a pipeline and a Step Functions state machine becomes a Flow. Much of what crawlers and small Lambda functions do around them is built into Trinium's nodes.
The shape of the move
- Each Glue job becomes a pipeline, built on a canvas instead of written as a Spark script, and run on your own server.
- The state machine becomes a Flow: a step per pipeline, side by side or in order, with wires for success and failure.
- Crawlers mostly disappear. Inputs show their columns as you build, partition folders become columns, and Iceberg tables written through the Glue Data Catalog (Enterprise) stay current for Athena without one.
- Most Lambda functions disappear too. Checking that a file can be read, moving files after a run, calling an API and sending an alert are things the nodes and Flows do.
- Data still moves through storage. One pipeline writes to S3 or a table, and the next reads it; the Flow makes sure the first has finished.
Piece by piece
AWS Glue
| Theirs | Trinium | Notes |
|---|---|---|
| Glue ETL job | No node | A pipeline, built on the canvas instead of written as a script. |
| Crawler | S3 Files, Apache Iceberg output | Not needed to read: every file and database input shows its columns as you build, and key=value folders become columns. Iceberg tables written through the Glue Data Catalog (Enterprise) are queryable in Athena without a crawler. |
| Data Catalog table | Apache Iceberg input, Apache Iceberg output, Athena | Iceberg reads and writes through a Glue catalog (Enterprise); Amazon Athena queries any table the catalog tracks. |
| Job bookmark | S3 Files, Postgres input | Bookmark mode on a cloud file input reads only new files; a database input's bookmark reads only rows past where the last successful run stopped. |
| ApplyMapping, DropFields, RenameField | Manage Columns | |
| Filter, Join, Union transforms | Filter, Join, Union | |
| Relationalize | JSON | |
| Trigger (scheduled) | No node | A schedule on a timer. |
| Workflow, conditional trigger | No node | A Flow: steps start when earlier ones succeed, fail or finish. |
AWS Step Functions
| Theirs | Trinium | Notes |
|---|---|---|
| Task state running a Glue job | No node | A Run pipeline step in a Flow, which waits for the pipeline to finish. |
| Task state running a Lambda function | API Lookup, API Batch Lookup, List Files | Usually not a step at all: checking files, calling an API and moving files are things the nodes do. A Lambda that computes something for rows keeps running where it is, behind a Function URL: API Batch Lookup sends it the rows. |
| Choice state | No node | A Flow branches on how a step ended (on success, on failure, on complete), not on values in its output. Inside a pipeline, Filter and Route branch on values. |
| Parallel state | No node | Steps with no wire between them never wait for each other, and run side by side on more than one server; a step with several incoming wires waits for all of them. |
| Map state | S3 Files | A cloud file input can start one run per file. A text parameter runs one pipeline against each region, customer or other source on a fixed list, each value as its own run. |
| Wait state | No node | Not supported. |
| Retry | No node | A failed run is never re-run automatically, so nothing is loaded or sent twice. Inside a run, a request that only reads is retried. |
| Catch | No node | An on failure wire to the step that should run instead. |
Around the pipeline
| Theirs | Trinium | Notes |
|---|---|---|
| Amazon SNS failure email | No node | A Notify step in the Flow, or the schedule's own notifications: email, Slack or Discord. |
| Amazon EventBridge rule on new S3 objects | No node | A schedule that runs when files arrive, checking the folder every few minutes. |
| Amazon Athena query | Athena | |
| Amazon S3 | S3 Files, S3 File | Including .gz files, partition folders and writing one folder per value of a column. |
What does not carry over
- Spark itself. Trinium runs a pipeline on one server, using all of it, and scales by running more pipelines on more servers. It does not split one job across a cluster.
- Wait states, loops, and branching on values in a step's output. A Flow branches on how a step ended.
- Automatic retries of a whole job. A failed run is reported and a person decides, so nothing is loaded or sent twice.
- Event-driven starts. A file-arrival schedule checks the folder every few minutes; it does not react to an S3 event.
Moving a job
- Create connections for the buckets and databases the job uses. On AWS, a shared connection can use the server's IAM role instead of keys.
- Rebuild the script's reads, transforms and writes as nodes, checking each node's columns as you go.
- Replace the crawler: read the folders directly, or write Iceberg tables through the Glue Data Catalog (Enterprise).
- Press Test for a 50-row sample, deploy to Staging, and compare a full run with the Glue output using Data Compare.
- Rebuild the state machine as a Flow, put it on a schedule, and add a Notify step where the state machine sent an alert.
AWS terms are from AWS's public documentation: AWS Glue concepts and Step Functions workflow states. AWS, AWS Glue, AWS Step Functions, AWS Lambda, Amazon S3, Amazon Athena, Amazon SNS and Amazon EventBridge are trademarks of Amazon.com, Inc. or its affiliates. Trinium is not affiliated with Amazon Web Services.