Databricks: Lakehouse Data Engineering, Step by Step
An eight-module course on Databricks: the lakehouse platform, Spark DataFrames, Delta Lake, Unity Catalog, Auto Loader ingestion, declarative pipelines, jobs and deployment, and SQL, performance, cost, and MLflow.
8 lessons · about 79 minutes of reading
Log in to track your progress through this course.
Start lesson 1- 1
Databricks Module 1: The Lakehouse Platform and Your First Notebook
Understand what Databricks is, how its workspace and compute fit together, and run your first notebook on Free Edition or a trial workspace.
9 min read
- 2
Databricks Module 2: Notebooks and Spark DataFrames with PySpark
Create DataFrames, select, filter, aggregate, join, and use Spark SQL in Databricks notebooks.
10 min read
- 3
Databricks Module 3: Delta Lake Tables, MERGE, and Time Travel
Create Delta tables, update data with MERGE, query earlier versions, and keep tables healthy with OPTIMIZE and VACUUM.
11 min read
- 4
Databricks Module 4: Unity Catalog, Volumes, and Governance
Organize data in catalogs and schemas, store files in volumes, grant permissions, and use lineage.
9 min read
- 5
Databricks Module 5: Ingesting Files with Auto Loader and COPY INTO
Load new files incrementally with Auto Loader, handle schema evolution, and use COPY INTO for simple batch loads.
10 min read
- 6
Databricks Module 6: Declarative Pipelines and the Medallion Architecture
Build bronze, silver, and gold tables declaratively with Lakeflow Spark Declarative Pipelines, including data quality expectations.
10 min read
- 7
Databricks Module 7: Jobs, Orchestration, and Deploying with Git and Bundles
Schedule multi-task workflows with Lakeflow Jobs, pass parameters, handle failures, and deploy code with Git and Asset Bundles.
10 min read
- 8
Databricks Module 8: SQL Warehouses, Performance, Cost, and a First Look at MLflow
Serve analysts with Databricks SQL, tune Spark and Delta performance, control cost, and track a model with MLflow.
10 min read