Databricks Module 7: Jobs, Orchestration, and Deploying with Git and Bundles

Step 1: Create a job with several tasks

A job is a set of tasks with dependencies. A task can run a notebook, a Python script or wheel, SQL, a pipeline update, or another job. Define the order by making one task depend on another. Run tasks on job compute, which starts for the run and shuts down afterward, or on serverless compute, and avoid all-purpose clusters for scheduled work because they cost more.

Job: daily_orders
  task 1: ingest_orders     (notebook)         Auto Loader into orders_bronze
  task 2: run_pipeline      (pipeline update)  bronze -> silver -> gold   depends on 1
  task 3: publish_report    (SQL)              refresh dashboard tables   depends on 2

Step 2: Pass parameters with widgets

Parameters make one notebook reusable across dates and environments. Define a widget in the notebook and set its value from the job's parameters. Read it with dbutils.widgets.get.

dbutils.widgets.text("run_date", "")
dbutils.widgets.text("catalog", "lab")

run_date = dbutils.widgets.get("run_date")
catalog = dbutils.widgets.get("catalog")

df = spark.table(catalog + ".sales.orders_silver").filter("date(ingested_at) = '" + run_date + "'")
print(df.count())

Step 3: Schedule, retry, and alert

Add a schedule with a cron expression and a time zone, or trigger runs when files arrive. Configure retries for transient failures, a timeout so a stuck task cannot run forever, and email or webhook notifications on failure. Keep tasks idempotent so a retry or a rerun for the same date gives the same result, for example by overwriting a date's partition or using MERGE.

Schedule     0 30 1 * * ?   (Quartz cron: every day at 01:30)
Retries      1-2 on each task
Timeout      set a maximum run time
Notify       on failure -> team email or webhook

Step 4: Keep code in Git

Do not develop only in the workspace. Connect a Git provider with Git folders, work on branches, review changes with pull requests, and run jobs from a specific branch or tag so production runs reviewed code. Move shared logic out of notebooks into Python modules that you can test.

Step 5: Deploy with Databricks Asset Bundles

Asset Bundles let you describe jobs, pipelines, and other resources as YAML files in your repository and deploy the same definition to dev and prod with one command. That makes environments consistent and changes reviewable. Check the current name and command reference, since tooling evolves.

# databricks.yml (simplified)
bundle:
  name: orders_project

resources:
  jobs:
    daily_orders:
      name: daily_orders
      tasks:
        - task_key: ingest_orders
          notebook_task:
            notebook_path: ./notebooks/ingest_orders.py

targets:
  dev:
    mode: development
    default: true
  prod:
    mode: production

# deploy and run
# databricks bundle validate
# databricks bundle deploy -t dev
# databricks bundle run daily_orders -t dev