Input node
BigQuery input
Read a table or SQL query from BigQuery. Columns AND what the query will scan both show without a run.
BigQueryWhat it does
Reads from Google BigQuery through a saved Connection and streams the result into the pipeline in batches. Pick a table with the live dataset and table browser, or write your own SQL. Before anything runs, the builder shows both the columns the query returns AND roughly how much data it will scan: BigQuery can plan a query without executing it, so this costs nothing. Credentials live on the Connection.
Inputs and outputs
| Inputs | None. This node starts a pipeline. |
| Outputs | One. |
When to use it
- Pull warehouse data into a pipeline for downstream work
- Push filtering and aggregation down to BigQuery so far less data crosses the wire
- Move a BigQuery result into a database, a file or a spreadsheet
Examples
Pick a table
analytics.events
The browser lists real datasets and tables.
Custom query
SELECT user_id, COUNT(*) AS hits FROM analytics.events GROUP BY user_id
Prune by partition
SELECT * FROM analytics.events WHERE event_date >= CURRENT_DATE() - 7
Filtering the partition column is the difference between scanning a week and scanning everything.
Tips
- Press ⤢ beside SQL query for the SQL Studio: a full-screen editor that browses the connection's schemas, tables and columns and runs your query with Run to here without closing.
- The scan estimate under the query box is the number you are billed on. If it looks enormous, add a partition filter or select fewer columns; SELECT * on a wide table is the usual culprit.
- Columns appear in the builder as soon as the query is valid, with no run needed, and the check itself scans nothing.
- This query is BigQuery SQL (GoogleSQL), which is different from Trinium's row expressions.
- Set the Location on the Connection unless your datasets are in the US; most 'dataset not found' errors are really a location mismatch.
- Numbers, booleans, dates and timestamps come across typed. NUMERIC and nested (STRUCT/ARRAY) columns arrive as text, so cast them in the query if you need them typed.