Flows
A Flow runs pipelines one after another, independently, or depending on how an earlier one ended. It is the layer above pipelines, where the order of whole jobs is decided.
Pipelines move rows; Flows move jobs
Inside a pipeline, a wire carries rows and every node runs at once. In a Flow, each step is a whole pipeline run, and a wire means "start this step after that one finishes, if it ended the way the wire says". That finish-then-start barrier is what a single streaming pipeline cannot express.
A Flow moves control, not data. When one pipeline's output is the next one's input, the first writes it somewhere both can reach, such as a table, a bucket or a folder, and the Flow makes sure it has finished before the second starts.
Steps and wires
- Run pipeline runs one pipeline and waits for it to finish.
- Notify sends a message to any number of people and notification destinations. A failed send never fails the Flow.
Each wire carries a condition: on success, on failure or on complete (either way).
- Two steps with no wire between them never wait for each other. Each server runs one production pipeline at a time, so they run side by side when there is more than one server.
- A pipeline runs once at a time in each environment, for each set of text parameter values. A step whose pipeline is already running with the same values (its own schedule, a person, another Flow) waits, shows why on the canvas, and starts when that run ends.
- A step whose pipeline runs each file separately starts one run per file, and ends Failed if any of them fails, naming the file.
- A pipeline step with several incoming wires runs only when every one of them is satisfied. If any cannot be, because its step ended the other way or was skipped, the step is skipped, and so is everything that depended on it.
- A notify step fires as soon as any one of its incoming wires is satisfied, so a notify step wired on failure from three steps means "tell me if any of them fails".
Environments and versions
A Flow run belongs to Staging or Production, chosen when you run it or schedule it. Every step runs the version its pipeline holds in that environment, so a Flow never runs a draft. Running a Flow in Production needs every step deployed to Production and Production access on every connection the steps use.
Every save that changes what a Flow does is kept as a new version, listed under Versions, and Restore brings an old one back as a new version. Moving cards around makes no version.
Parameters
A Flow can declare parameters of its own, asked for when it runs. Each step says exactly which values it hands its pipeline, written on the step: a value is never passed along just because two names match. A date value such as ${date-1} reads the moment the Flow run was queued, so every step of one Flow run reads the same day.
Scheduling and stopping
A Flow is scheduled like a pipeline, on a timer. A scheduled Flow never overlaps itself: if its last run is still going when the next time comes, that time is skipped. Cancel stops the steps that are running and marks the rest cancelled.
What a Flow does not do
A Flow does not loop, and a step cannot hand a value it produced to a later step: hand data over through storage instead. Building Flows needs the Engineer role or above (see Roles and workspaces).