Input node
Parquet Files
Read one or more columnar Apache Parquet files.
What it does
Reads one or more Apache Parquet files and streams their rows into the pipeline. Parquet stores its own schema, so there is nothing to infer and nothing to configure: every column arrives with exactly the type it was written with. Several files are read in order and stacked into one stream.
Inputs and outputs
| Inputs | None. This node starts a pipeline. |
| Outputs | One. |
When to use it
- Load columnar data from a data lake or a previous Trinium run
- Move a large dataset efficiently, since Parquet is compressed and typed
- Read a folder of part-files written by another job as one table
Examples
Single file
events.parquet
Columns and types come straight from the file's footer.
Stacked part-files
part-0.parquet, part-1.parquet, part-2.parquet
Read in order and concatenated into one stream.
Track the source
Add source file name as column: source_file
Each row is tagged with the file it came from, useful once several are stacked.
A partner added a column
If a file is different: Reconcile it
The extra column is dropped, the run continues, and the Messages tab names the file.
Tips
- Types are authoritative: no inference and no overrides, unlike CSV. This is the one file format whose columns are never guessed.
- The first file whose footer can be read sets the columns, and later files are matched to it BY NAME. A missing column is filled with nulls, an extra one is dropped, and a file with no columns in common is skipped. Every difference is named in the run's Messages.
- A file that is not really Parquet, or is cut short, is skipped and named, wherever it sorts. Under Stop the run, it stops the run. If no file can be read, the run fails and names the file.
- A column the first file declares REQUIRED cannot be filled with nulls, so a later file missing it is skipped and the column is named. Most tools write columns as optional, so this is rare.
- Change that with “If a file is different”: Skip reads only files matching the first exactly, and Stop the run fails on any difference.
- A column that first appears in file 5 is dropped, not picked up: a source has to declare its columns before it emits a row, so only the first file can define them.
- Its columns show in the builder without a run, read from the first file's footer without touching a single row.
- Ideal as the handover format between Trinium pipelines: it's the only file format here that carries types.
- Turn on 'Run each file separately' to process every file as its own run, handy when one bad file shouldn't fail the rest. Every start does this, a schedule and a flow step included. A Test run still reads them together. The runs take turns, one at a time.