Databricks Module 1: The Lakehouse Platform and Your First Notebook

Step 1: Understand the lakehouse idea

Traditionally, raw data went into a data lake and curated data into a warehouse, which meant copies and two sets of tools. A lakehouse keeps data once, in open table formats such as Delta Lake on cloud storage, and adds transactions, governance, and fast SQL on top. Databricks created Delta Lake and builds its platform around it.

Cloud storage (S3 / ADLS / GCS)  +  Delta Lake tables  +  Unity Catalog governance
        |
   One copy of data  ->  SQL analytics, data engineering, data science, ML

Step 2: Know the main building blocks

The workspace holds notebooks, files, and dashboards. Compute runs your code: serverless compute starts quickly and is managed for you, and classic clusters give you control over machine types and Spark settings. Unity Catalog manages data and permissions. Lakeflow covers ingestion, declarative pipelines, and job orchestration. Databricks SQL provides SQL warehouses and dashboards for analysts.

Workspace        notebooks, repos, dashboards, jobs
Compute          serverless, or classic all-purpose and job clusters
Unity Catalog    catalogs, schemas, tables, volumes, permissions, lineage
Lakeflow         Connect (ingest), Spark Declarative Pipelines (transform), Jobs (orchestrate)
Databricks SQL   SQL warehouses, queries, dashboards
MLflow           experiment tracking and model management

Step 3: Get a workspace to practice in

Databricks Free Edition is a free way to learn, and it replaced the older Community Edition. It has limits and does not include every feature of paid workspaces, so check what it supports. Otherwise, use a trial or your company workspace. Paid compute is billed in Databricks Units plus cloud costs, so stop or terminate compute when you finish, and never leave clusters running.

Step 4: Create a notebook and run it

Create a notebook, choose Python as the default language, and attach it to compute. Cells run with Shift and Enter. The spark object (a SparkSession) is already available, and display shows results as an interactive table.

df = spark.range(1, 11).withColumnRenamed("id", "n")
display(df)

print(spark.version)
print(df.selectExpr("sum(n) as total").collect())

Step 5: Mix languages in one notebook

A magic command at the top of a cell switches its language, so you can use SQL, Python, and Markdown together. Use the Python cell to create data and the SQL cell to query it.

# Python cell
spark.range(1, 6).createOrReplaceTempView("numbers")

%sql
SELECT id, id * id AS square FROM numbers ORDER BY id;

%md
## Notes
Markdown cells document your steps.