Pipelines and the canvas

A pipeline is a set of nodes wired together on a canvas. Each node does one job, and rows stream from node to node as they arrive.

Nodes and wires

A node reads data, changes it, or writes it out. You drag nodes onto the canvas from the palette, configure each one in the pane beside the canvas, and wire an output anchor to an input anchor. Some nodes have more than one output: a Filter sends matching rows one way and the rest another, and a Join keeps the unmatched rows of each side on outputs of their own.

When a pipeline runs, every node starts at once. Rows travel between them in batches, so the first rows can reach the output while the source is still being read, and a pipeline uses the whole machine rather than one node at a time. Nodes that must see everything first, such as a sort, a join or a summary, hold their rows within a memory budget and spill to disk when it runs short, so a job larger than memory can still finish.

Columns before you run

The builder works out every node's output columns as you build, without running the pipeline. A source shows the columns of the table, query or file you pick; each node after it shows what it will produce. A node with a setting that will not work, such as a column that does not exist or an expression with a typo, is flagged on the canvas before you run.

Every rule you type, in a filter, a formula or a join condition, is written in one SQL expression language.

Working on the canvas

• Left-drag selects, right-drag pans, and the wheel zooms.
• Nodes can be copied and pasted within a pipeline, and every change can be undone.
• A Container groups nodes and can be turned off to bypass them; a Note labels part of the canvas. Both are canvas tools, not nodes.
• Save writes your draft. Running never saves: only Save does.

Seeing the data

A Test run runs the pipeline on a sample. The dock under the canvas then shows each node's rows (Data), columns and types (Metadata) and anything the node had to say (Messages).

Next

A saved draft is not what schedules run. Versions, Staging and Production explains how a draft becomes a version and reaches each environment.