Building a Data Lake: Step by Step

A seven-module course on designing and building a data lake on AWS: architecture, S3 storage, file formats and partitioning, ingestion, cataloging and querying, open table formats, and governance.

7 lessons · about 65 minutes of reading

Log in to track your progress through this course.

1. 1

   Building a Data Lake Module 1: What a Data Lake Is and How It Is Organized

   Understand data lakes, warehouses, and lakehouses, and learn the raw, clean, and curated layer design.

   8 min read

2. 2

   Building a Data Lake Module 2: Storage Foundations on Amazon S3

   Create buckets and a clean folder structure, and turn on encryption, versioning, and lifecycle rules.

   9 min read

3. 3

   Building a Data Lake Module 3: File Formats, Compression, and Partitioning

   Choose between CSV, JSON, Parquet, and Avro, and design partitions that make queries fast and cheap.

   9 min read

4. 4

   Building a Data Lake Module 4: Ingesting Data into the Lake

   Bring in data from files, databases, applications, and streams, and make ingestion repeatable and safe to rerun.

   9 min read

5. 5

   Building a Data Lake Module 5: Cataloging and Querying with Glue and Athena

   Register data in the AWS Glue Data Catalog and query it in place with SQL in Amazon Athena.

   10 min read

6. 6

   Building a Data Lake Module 6: Open Table Formats with Iceberg, Delta Lake, and Hudi

   Add transactions, updates, deletes, and time travel to lake data with Apache Iceberg, Delta Lake, or Apache Hudi.

   10 min read

7. 7

   Building a Data Lake Module 7: Governance, Security, Data Quality, and Cost

   Protect data with IAM and Lake Formation, add quality checks and lineage, and keep the lake affordable.

   10 min read