Databricks input

Read a table or SQL query from a Databricks SQL warehouse. Results stream back columnar.

Category

Input

Connection

Databricks

What it does

Runs a query on a Databricks SQL warehouse through a saved Connection and streams the result into the pipeline. Pick a table with the live catalog, schema and table browser, or write your own SQL. Results come back in the pipeline's own columnar format, so nothing is converted row by row on the way in. Credentials live on the Connection.

Inputs and outputs

When to use it

• Pull lakehouse tables into a pipeline for downstream work
• Push filtering and aggregation down to the warehouse so less data crosses the wire
• Move Databricks data into a database, a file or another warehouse

Examples

main.sales.orders

SELECT id, total FROM main.sales.orders WHERE order_date >= current_date() - 7

SELECT region, SUM(total) AS revenue FROM main.sales.orders GROUP BY region

Tips

• Press ⤢ beside SQL query for the SQL Studio: a full-screen editor that browses the connection's schemas, tables and columns and runs your query with Run to here without closing.
• Columns appear when you press Refresh columns: Trinium asks the warehouse for the query with no rows. That can start a paused warehouse, so it only happens when you press it. The answer is kept while you edit the pipeline, until you change this node.
• This query is Databricks SQL, which is different from Trinium's row expressions.
• Types come across natively: numbers, dates, timestamps and decimals stay typed rather than arriving as text.
• The warehouse has to be running (or able to auto-start) before a run can read from it; a cold warehouse mostly shows up as a slow first run.
• There is no driver to install: this talks to the warehouse over its own API.

Related nodes