What is the difference between Delta Lake and Parquet?

Delta Lake stores data as Parquet files plus a transaction log. The log gives ACID transactions, updates, deletes, schema enforcement, and time travel, which plain Parquet folders lack.

Databricks Module 3: Delta Lake Tables, MERGE, and Time Travel

Introduction

Delta Lake is the storage layer at the heart of Databricks. It adds ACID transactions, updates and deletes, schema enforcement, and time travel to files in cloud storage. Tables you create in Databricks use Delta by default. This module works with Delta tables hands-on.

Prerequisites

Module 2 completed
A catalog and schema you can write to (for example main.default)

Step-by-Step Guide

Step 1: Create a Delta table

Create a table with SQL or write a DataFrame to a table. Tables in Unity Catalog have three-level names: catalog, schema, then table. Replace main.default with a catalog and schema you have permission to use.

CREATE TABLE IF NOT EXISTS main.default.customers (
  customer_id BIGINT,
  name STRING,
  city STRING,
  updated_at TIMESTAMP
);

INSERT INTO main.default.customers VALUES
  (1, 'Asha', 'Pune', current_timestamp()),
  (2, 'Ravi', 'Delhi', current_timestamp());

SELECT * FROM main.default.customers;

Step 2: Update and delete rows

Unlike plain Parquet files, Delta tables support UPDATE and DELETE. Each change is a transaction recorded in the table's log, so readers never see half-finished work.

UPDATE main.default.customers SET city = 'Pune City' WHERE customer_id = 1;
DELETE FROM main.default.customers WHERE customer_id = 2;

Step 3: Upsert with MERGE

MERGE applies inserts and updates from a source to a target in one transaction. It is the standard way to load changed records, such as a daily extract from a source system, without creating duplicates.

CREATE OR REPLACE TEMP VIEW customer_updates AS
SELECT * FROM VALUES
  (1, 'Asha Rao', 'Mumbai', current_timestamp()),
  (3, 'Meera',    'Chennai', current_timestamp())
AS t(customer_id, name, city, updated_at);

MERGE INTO main.default.customers AS t
USING customer_updates AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Step 4: Use history and time travel

Every change creates a new table version. DESCRIBE HISTORY lists them, and you can query an earlier version by number or timestamp, or restore the table. Time travel helps with audits, debugging, and undoing a bad load. History is kept only for a retention period, so do not treat it as a long-term backup.

DESCRIBE HISTORY main.default.customers;

SELECT * FROM main.default.customers VERSION AS OF 1;
SELECT * FROM main.default.customers TIMESTAMP AS OF '2026-10-16T10:00:00';

RESTORE TABLE main.default.customers TO VERSION AS OF 1;

Step 5: Maintain tables

Many small writes create many small files. OPTIMIZE compacts them for faster reads, and VACUUM removes data files no longer referenced by the table after the retention period. Modern Databricks can run maintenance automatically for managed tables (predictive optimization), so check whether it is enabled before scheduling your own. Liquid clustering, set with CLUSTER BY, organizes data for the columns you filter on and replaces older partitioning approaches for many tables.

OPTIMIZE main.default.customers;
VACUUM main.default.customers;

CREATE TABLE main.default.events (
  event_id BIGINT, user_id BIGINT, event_time TIMESTAMP, action STRING
) CLUSTER BY (event_time);

FAQs

What is the difference between Delta Lake and Parquet?: Delta Lake stores data as Parquet files plus a transaction log. The log gives ACID transactions, updates, deletes, schema enforcement, and time travel, which plain Parquet folders lack.

Summary

Delta tables add transactions to cloud storage files. Use UPDATE, DELETE, and MERGE for changes, DESCRIBE HISTORY and time travel to inspect or restore, and OPTIMIZE, VACUUM, and clustering to keep tables fast.