Why use a pipeline instead of notebooks and jobs?

Pipelines manage dependencies, incremental processing, retries, and quality metrics for you, so you write the logic and the platform does the plumbing. Notebooks and jobs are still better for custom logic that does not fit the table-by-table model.

Databricks Module 6: Declarative Pipelines and the Medallion Architecture

Introduction

Instead of writing every step of a pipeline and its orchestration, a declarative pipeline lets you define the tables you want and how each is computed, and Databricks handles ordering, incremental processing, and recovery. This framework was called Delta Live Tables, then Lakeflow Declarative Pipelines, and was renamed Lakeflow Spark Declarative Pipelines in late 2025. The syntax and UI keep evolving, so follow the current documentation for exact keywords.

Prerequisites

Module 5 completed
A workspace that supports pipelines (check Free Edition limits)

Step-by-Step Guide

Step 1: Understand the medallion architecture

Bronze tables hold raw data as ingested. Silver tables hold cleaned, validated, deduplicated data. Gold tables hold business-level aggregates ready for dashboards and models. Declaring each layer as a table in a pipeline gives you clear lineage and lets the engine process only new data at each step.

Files --> orders_bronze (raw) --> orders_silver (clean) --> daily_revenue_gold (aggregate)

Step 2: Define a bronze streaming table

A streaming table processes new data incrementally. The SQL below reads files with Auto Loader. Create a pipeline in the workspace, add a SQL file with these statements, and point the pipeline at a target catalog and schema.

CREATE OR REFRESH STREAMING TABLE orders_bronze
COMMENT 'Raw orders as ingested'
AS SELECT *, _metadata.file_path AS source_file, current_timestamp() AS ingested_at
FROM STREAM read_files(
  '/Volumes/lab/sales/raw_files/orders/',
  format => 'json'
);

Step 3: Add a silver table with expectations

Expectations are data quality rules attached to a table. Each rule is a condition, and you choose what happens to violating rows: keep them but record the violation (the default), drop them, or fail the update. Dropped and failed counts appear in the pipeline's quality metrics.

CREATE OR REFRESH STREAMING TABLE orders_silver (
  CONSTRAINT valid_order_id EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT positive_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW,
  CONSTRAINT known_status EXPECT (status IN ('SHIPPED', 'PENDING', 'CANCELLED'))
)
COMMENT 'Cleaned orders'
AS SELECT order_id, customer_id, CAST(amount AS DECIMAL(12,2)) AS amount, status, ingested_at
FROM STREAM orders_bronze;

Step 4: Add a gold materialized view

A materialized view stores the result of a query and keeps it up to date, recomputing incrementally where possible. Use one for aggregates that dashboards read often.

CREATE OR REFRESH MATERIALIZED VIEW revenue_by_status
COMMENT 'Revenue per order status'
AS SELECT status, count(*) AS orders, sum(amount) AS revenue
FROM orders_silver
GROUP BY status;

Step 5: Run the pipeline and read the results

Run the pipeline from the UI. It builds a graph of dependencies, runs each step in order, and shows row counts and quality results. Use a triggered run for scheduled batch updates, or continuous mode for low latency, remembering that continuous mode keeps compute running and costs more. Query the resulting tables like any other Delta table.

SELECT * FROM lab.sales.revenue_by_status;

FAQs

Why use a pipeline instead of notebooks and jobs?: Pipelines manage dependencies, incremental processing, retries, and quality metrics for you, so you write the logic and the platform does the plumbing. Notebooks and jobs are still better for custom logic that does not fit the table-by-table model.

Summary

Declare bronze streaming tables, silver tables with expectations, and gold materialized views in a pipeline, and let Databricks handle dependencies and incremental processing. Check current names and syntax, since this feature has been renamed more than once.