Apache Iceberg output

Write rows to an Apache Iceberg table through AWS Glue, S3 Tables or a REST catalog, appending or fully refreshing, with auto-create and partitioning. No crawler needed.

Category

Output

Connection

Apache Iceberg

Edition

Enterprise

What it does

Writes the incoming rows into an Apache Iceberg table through the AWS Glue Data Catalog, Amazon S3 Tables or an Iceberg REST catalog, and commits them as ONE atomic snapshot. Appends, or overwrites (a full refresh). The table is auto-created from the incoming columns (with an optional identity partition spec) when it doesn't exist. No crawler ever runs: the write IS the catalog update.

Inputs and outputs

When to use it

• Land pipeline output in a lakehouse table your catalog tracks
• Replace a table's data each run (overwrite / full refresh)
• Create a partitioned Iceberg table straight from a transform

Examples

namespace: sales · table: orders · Mode: append

Mode: overwrite

Partition by: region, day

Tips

• Append is atomic: a mid-run failure commits nothing and leaves the table untouched.
• Overwrite writes the new rows into a table of their own first and swaps it in only once they are committed, so a failed run leaves the old table exactly as it was. Snapshot history restarts, so use append to keep time-travel.
• After an overwrite the old table's data files are deleted, if the catalog allows it. A catalog that refuses keeps them, and the run's Messages say so; Polaris refuses unless its admin allows purge.
• Each overwrite moves the table's storage folder, because a catalog keeps a table in the folder of the name it was created under. Engines reading through the catalog never notice; anything that reads the folder by path will.
• Partition by only shapes a table this node creates. Appending to a table that already exists follows that table's own partitioning, including a hidden one such as day(ts), and a Partition by that disagrees with it stops the run and says so.
• A new table stores timestamps in microseconds, Iceberg's precision, so a nanosecond value loses its last three digits. A timestamp with a time zone is stored in UTC, as the same instant.
• Data files are compressed with the table's write.parquet.compression-codec, or zstd when the table names none.
• Appending to a table that already exists matches your columns to the table's by name, so their order doesn't matter. A column the table doesn't have stops the run and is named; a column your rows lack is left empty, unless the table requires it.
• Every snapshot it commits records the pipeline and run that wrote it (trinium.pipeline-id and trinium.run-id in the snapshot summary), so any engine reading the table can see where its data came from.
• Written through the AWS Glue Data Catalog, the table can be queried from Athena as soon as the run finishes: the commit is the catalog update, so there is no crawler and no MSCK REPAIR TABLE.
• The namespace must already exist: a Glue database, an S3 Tables namespace or a REST catalog namespace. The node creates tables, never namespaces.
• Point it at the AWS Glue Data Catalog, an S3 Tables bucket, or any Iceberg REST catalog your team already runs.

Related nodes