AWS Glue Module 8: Monitoring, Performance, Data Quality, and Cost

Step 1: Monitor runs with logs and metrics

Each run writes logs to Amazon CloudWatch Logs, with separate streams for output and errors. Job run details in Glue show status, duration, and DPU usage. Enable job metrics and the Spark UI for deeper inspection of stages, skew, and shuffles. Create CloudWatch alarms or EventBridge rules for failed runs.

aws logs tail /aws-glue/jobs/error --since 1h
aws glue get-job-run --job-name daily-revenue-job --run-id jr_0123456789 \
  --query "JobRun.[JobRunState,ExecutionTime,DPUSeconds,ErrorMessage]"

Step 2: Pick worker types and use auto scaling

Workers differ in memory and cores (for example G.1X and G.2X for general work, with larger and memory-optimized options). Start small, check whether the job is memory-bound or has idle workers, and scale up only when needed. Auto scaling lets Glue add and remove workers during a run up to a maximum you set, so you stop paying for idle capacity. Check the current worker types and Flex execution options in the documentation.

Step 3: Fix common performance problems

Typical causes of slow jobs: reading too many small files (use groupFiles), scanning all partitions (use pushdown predicates), data skew where one key has most rows, large shuffles from wide joins (broadcast the small table), and many tiny output files. Use the Spark UI to find the slow stage before changing anything.

Step 4: Add data quality checks with Glue Data Quality

Glue Data Quality evaluates rules written in the Data Quality Definition Language (DQDL) against a table or a DynamicFrame, produces a score and per-rule results, and can stop the job or route bad rows. Add a check before you publish curated data.

Rules = [
    RowCount > 0,
    IsComplete "order_id",
    IsUnique "order_id",
    ColumnValues "amount" >= 0,
    ColumnValues "status" in ["SHIPPED", "PENDING", "CANCELLED"]
]

Step 5: Control cost and clean up

Costs come from job and crawler run time, interactive sessions, and Data Catalog usage beyond the free tier. Right-size workers, use auto scaling, set job timeouts, schedule crawlers only when needed, process incrementally with bookmarks, and stop interactive sessions when idle. When you finish the course, delete the lab resources so nothing keeps billing.

aws glue delete-trigger --name nightly-daily-revenue
aws glue delete-workflow --name orders-pipeline
aws glue delete-job --job-name daily-revenue-job
aws glue delete-crawler --name orders-raw-crawler
aws glue delete-database --name glue_lab
aws s3 rm s3://my-glue-lab-123456 --recursive
aws s3api delete-bucket --bucket my-glue-lab-123456
aws iam delete-role-policy --role-name GlueLabRole --policy-name GlueLabS3Access
aws iam detach-role-policy --role-name GlueLabRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
aws iam delete-role --role-name GlueLabRole