Are Glue triggers in my local time zone?

No. Schedules use cron expressions evaluated in UTC, so convert your local time when choosing the hour.

AWS Glue Module 7: Triggers, Workflows, and Job Parameters

Introduction

A real pipeline runs on a schedule, in order, and recovers from failures. Glue provides triggers and workflows for simple pipelines, and works with Amazon EventBridge and AWS Step Functions for richer orchestration. This module shows how to schedule, chain, and parameterize jobs.

Prerequisites

Module 6 completed

Step-by-Step Guide

Step 1: Pass parameters to a job

Hard-coded paths and dates make a job rigid. Define parameters as job arguments that start with two dashes, read them with getResolvedOptions, and override them per run. This lets one script serve dev and prod, or process a specific date.

# in the script
args = getResolvedOptions(sys.argv, ["JOB_NAME", "source_path", "target_path", "run_date"])
source_path = args["source_path"]

# start a run with overrides
aws glue start-job-run --job-name daily-revenue-job --arguments '{
  "--source_path": "s3://my-glue-lab-123456/raw/orders/",
  "--target_path": "s3://my-glue-lab-123456/curated/daily_revenue/",
  "--run_date": "2026-09-26"
}'

Step 2: Schedule a job with a trigger

A scheduled trigger starts a job at times given by a cron expression, in UTC. Create the trigger, then activate it. Triggers can also be on demand or conditional on another job or crawler finishing.

aws glue create-trigger --name nightly-daily-revenue --type SCHEDULED \
  --schedule "cron(30 1 * * ? *)" \
  --actions JobName=daily-revenue-job --start-on-creation

Step 3: Chain steps with a workflow

A workflow groups triggers, crawlers, and jobs into a single pipeline you can run and monitor as a unit, such as crawl the raw data, run the ETL job, then crawl the curated output. Conditional triggers start the next step only when the previous one succeeds.

aws glue create-workflow --name orders-pipeline

aws glue create-trigger --name start-pipeline --workflow-name orders-pipeline \
  --type SCHEDULED --schedule "cron(0 1 * * ? *)" \
  --actions CrawlerName=orders-raw-crawler

aws glue create-trigger --name after-crawl --workflow-name orders-pipeline --type CONDITIONAL \
  --predicate '{"Conditions":[{"LogicalOperator":"EQUALS","CrawlerName":"orders-raw-crawler","CrawlState":"SUCCEEDED"}]}' \
  --actions JobName=daily-revenue-job

Step 4: Choose the right orchestrator

Glue workflows are simple and live inside Glue. Use AWS Step Functions when you need branching, error handling, waiting, and calls to many services. Use Amazon EventBridge to start jobs when files arrive or on rules. Use Amazon Managed Workflows for Apache Airflow if your team already uses Airflow. Pick the simplest tool that meets your needs.

Step 5: Handle failures and retries

Set the job's maximum retries for transient failures, and a timeout so a stuck job does not run up cost. Send failure alerts through EventBridge rules on job state change events to an SNS topic. Make jobs idempotent so a retry or a rerun produces the same result, for example by overwriting a date partition instead of appending.

aws glue update-job --job-name daily-revenue-job --job-update '{
  "Role": "GlueLabRole",
  "Command": {"Name": "glueetl", "ScriptLocation": "s3://my-glue-lab-123456/scripts/daily_revenue.py", "PythonVersion": "3"},
  "MaxRetries": 1,
  "Timeout": 60
}'

FAQs

Are Glue triggers in my local time zone?: No. Schedules use cron expressions evaluated in UTC, so convert your local time when choosing the hour.

Summary

Parameterize jobs with getResolvedOptions, schedule with triggers, chain crawlers and jobs in workflows, use Step Functions or EventBridge for richer needs, and configure retries, timeouts, alerts, and idempotent writes.