AWS Glue: ETL and the Data Catalog, Step by Step
An eight-module course on AWS Glue: setup, the Data Catalog and crawlers, visual and script ETL jobs, transforms, bookmarks and partitions, orchestration, and monitoring, quality, and cost.
8 lessons · about 75 minutes of reading
Log in to track your progress through this course.
Start lesson 1- 1
AWS Glue Module 1: What Glue Is and Setting It Up
Learn the parts of AWS Glue, how it is priced, and create the IAM role and S3 sample data you need for the course.
9 min read
- 2
AWS Glue Module 2: The Data Catalog and Crawlers
Create a catalog database, run a crawler over S3 data, and understand how tables, schemas, and partitions are recorded.
9 min read
- 3
AWS Glue Module 3: Your First ETL Job with Glue Studio
Build a visual ETL job that reads the cataloged table, filters and renames columns, and writes Parquet to S3.
9 min read
- 4
AWS Glue Module 4: Writing PySpark Scripts with GlueContext and DynamicFrames
Write a Glue ETL script by hand: read from the catalog, transform with DynamicFrames, and write partitioned Parquet.
10 min read
- 5
AWS Glue Module 5: Transforms and Cleaning Data
Use ApplyMapping, ResolveChoice, DropNullFields, Filter, and Join, and clean data with Spark functions.
10 min read
- 6
AWS Glue Module 6: Partitions, Pushdown Predicates, and Job Bookmarks
Process only new or needed data with partition filters and job bookmarks, and write well-partitioned output.
9 min read
- 7
AWS Glue Module 7: Triggers, Workflows, and Job Parameters
Schedule and chain crawlers and jobs, pass parameters, and handle failures and retries.
9 min read
- 8
AWS Glue Module 8: Monitoring, Performance, Data Quality, and Cost
Monitor jobs, tune workers and auto scaling, add Glue Data Quality rules, and keep Glue costs under control.
10 min read