Databricks Module 4: Unity Catalog, Volumes, and Governance
Introduction
Unity Catalog is Databricks' governance layer: one place to organize data, control who can see it, and track where it comes from, across workspaces. Understanding its structure early saves a lot of rework. Some governance features need a paid workspace or admin rights, so check what your account allows.
Prerequisites
- Module 3 completed
Step-by-Step Guide
Step 1: Learn the three-level namespace
Everything is addressed as catalog.schema.object. A catalog usually represents an environment or business domain, a schema groups related objects, and objects are tables, views, volumes, functions, and models. Common patterns are catalogs per environment (dev, prod) or per domain, with schemas for layers such as bronze, silver, and gold.
catalog prod
schema sales_bronze raw ingested tables
schema sales_silver cleaned tables
schema sales_gold business-ready tables
table orders_daily prod.sales_gold.orders_dailyStep 2: Create catalogs and schemas
If you have the privileges, create objects with SQL. Use a clear naming convention and put a comment on each so others know its purpose.
CREATE CATALOG IF NOT EXISTS lab COMMENT 'Course practice catalog';
CREATE SCHEMA IF NOT EXISTS lab.sales COMMENT 'Sales practice data';
USE CATALOG lab;
USE SCHEMA sales;
SELECT current_catalog(), current_schema();Step 3: Store files in volumes
Tables hold structured data. Volumes hold files, such as CSV, JSON, images, and libraries, governed by the same permissions. You read and write them with paths that start with /Volumes, for example the files an ingestion process will pick up.
CREATE VOLUME IF NOT EXISTS lab.sales.raw_files;
-- path: /Volumes/lab/sales/raw_files/
-- Upload a file in the UI, then read it:
df = spark.read.option("header", True).csv("/Volumes/lab/sales/raw_files/orders.csv")
display(df)Step 4: Grant permissions with SQL
Permissions are granted to users, groups, or service principals on securable objects. Users also need USE privileges on the parent catalog and schema to reach an object. Grant to groups instead of individuals, give the minimum needed, and use views to expose only certain columns or rows.
GRANT USE CATALOG ON CATALOG lab TO `data-analysts`;
GRANT USE SCHEMA ON SCHEMA lab.sales TO `data-analysts`;
GRANT SELECT ON TABLE lab.sales.orders_daily TO `data-analysts`;
SHOW GRANTS ON TABLE lab.sales.orders_daily;Step 5: Understand managed tables, external tables, and lineage
Managed tables store their data in the catalog's managed location, and Databricks handles file layout and cleanup, so dropping one removes its data. External tables point to data in storage you control, so dropping one leaves the files. Unity Catalog also records lineage automatically: which tables and columns feed which, which helps with impact analysis and audits. Open a table's Lineage tab in the catalog explorer to see it.
FAQs
- Why do I get a permission error on a table I can see?
- Access needs privileges at each level. You need USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT on the table. Ask an owner or admin to grant what is missing.
Summary
Unity Catalog organizes data as catalog.schema.object, stores files in volumes, controls access with grants to groups, and tracks lineage. Prefer managed tables, grant minimal privileges, and use clear naming and comments.
Related tutorials
- Databricks Module 8: SQL Warehouses, Performance, Cost, and a First Look at MLflow
- Databricks Module 7: Jobs, Orchestration, and Deploying with Git and Bundles
- Databricks Module 6: Declarative Pipelines and the Medallion Architecture
- Databricks Module 5: Ingesting Files with Auto Loader and COPY INTO
- Databricks Module 3: Delta Lake Tables, MERGE, and Time Travel
Comments
Log in or create a free account to join the discussion.