Laziness lets Spark see your whole chain of transformations and optimize it, for example by reading only the columns it needs, before running anything. Actions such as count, show, collect, and write trigger execution.

Databricks Module 2: Notebooks and Spark DataFrames with PySpark

Introduction

Databricks runs on Apache Spark, whose main data structure is the DataFrame: a distributed table of rows and named columns. You describe what you want and Spark plans and runs it in parallel. This module covers the PySpark operations you will use every day.

Prerequisites

Module 1 completed

Step-by-Step Guide

Step 1: Create a DataFrame

Create small DataFrames from Python data for practice, and read files for real work. Spark transformations are lazy: nothing runs until an action such as show, count, or write asks for a result.

from pyspark.sql import functions as F

orders = spark.createDataFrame(
    [(1, 101, "2026-09-01", 120.5, "SHIPPED"),
     (2, 102, "2026-09-01", 80.0,  "SHIPPED"),
     (3, 101, "2026-09-02", 45.25, "CANCELLED"),
     (4, 103, "2026-09-02", 300.0, "SHIPPED")],
    ["order_id", "customer_id", "order_date", "amount", "status"],
)
orders = orders.withColumn("order_date", F.to_date("order_date"))
orders.printSchema()
display(orders)

Step 2: Select, filter, and add columns

select picks columns, filter (or where) keeps rows, and withColumn creates or replaces a column. Reference columns with the F.col function or the col string, and combine conditions with & and | inside parentheses.

shipped = (orders
    .filter((F.col("status") == "SHIPPED") & (F.col("amount") > 50))
    .withColumn("amount_with_tax", F.round(F.col("amount") * 1.18, 2))
    .select("order_id", "customer_id", "amount", "amount_with_tax"))
display(shipped)

Step 3: Aggregate with groupBy

groupBy followed by agg computes summaries per group. Use alias to name results. Aggregations shuffle data between machines, which is the main cost in Spark, so filter before you group.

summary = (orders
    .groupBy("status")
    .agg(F.count("*").alias("orders"),
         F.sum("amount").alias("revenue"),
         F.avg("amount").alias("avg_order")))
display(summary.orderBy(F.desc("revenue")))

Step 4: Join DataFrames

join combines two DataFrames on a condition. Specify the join type (inner, left, right, full) and be careful about duplicate column names. Spark automatically broadcasts small tables to avoid a shuffle, and you can hint it for a small lookup table.

customers = spark.createDataFrame(
    [(101, "Asha", "Pune"), (102, "Ravi", "Delhi"), (103, "Meera", "Mumbai")],
    ["customer_id", "name", "city"],
)

joined = orders.join(F.broadcast(customers), "customer_id", "left")
display(joined.select("order_id", "name", "city", "amount"))

Step 5: Use Spark SQL on the same data

Register a DataFrame as a temporary view and query it with SQL. DataFrames and SQL are two ways to express the same plan, so use whichever reads better, and mix them freely. Check the query plan with explain when something is slow.

orders.createOrReplaceTempView("orders")

result = spark.sql("""
    SELECT order_date, count(*) AS orders, sum(amount) AS revenue
    FROM orders
    WHERE status <> 'CANCELLED'
    GROUP BY order_date
    ORDER BY order_date
""")
display(result)
result.explain()

FAQs

Why is Spark lazy?: Laziness lets Spark see your whole chain of transformations and optimize it, for example by reading only the columns it needs, before running anything. Actions such as count, show, collect, and write trigger execution.

Summary

Spark DataFrames are lazy distributed tables. Use select, filter, withColumn, groupBy and agg, and join (with broadcast for small tables), and use Spark SQL on temporary views when it reads better.