Building a Data Lake Module 1: What a Data Lake Is and How It Is Organized
Introduction
A data lake is a central store that holds large amounts of data in its original form, cheaply, until someone needs it. Unlike a database, it does not force a structure up front. This first module of the seven-module Building a Data Lake course explains what a lake is, how it differs from a warehouse, and the layered design that keeps one organized.
Prerequisites
- Basic SQL and cloud storage concepts
Step-by-Step Guide
Step 1: Know the three kinds of data store
A data warehouse stores cleaned, structured data in tables designed for fast reporting. A data lake stores any data (files, logs, images, JSON) cheaply on object storage and applies structure when the data is read. A lakehouse adds table features, such as transactions and schema enforcement, on top of lake storage so one copy of data serves both reporting and data science.
Warehouse structured tables, schema first, fast SQL, higher cost per TB
Lake any file format, schema when reading, cheap object storage
Lakehouse lake storage + table formats (Iceberg, Delta, Hudi) = SQL tables on filesStep 2: Decide when you need a lake
A lake helps when you have many data sources, large or varied data (logs, events, documents), data scientists who need raw data, or a need to keep history cheaply. It is a poor fit for a small company with a handful of clean tables: a managed database or warehouse is simpler. Many teams use a lake for storage and a warehouse for the polished reporting layer.
Step 3: Organize data in layers
The most common design uses zones, often called the medallion architecture. Raw (bronze) data is stored exactly as it arrived and never changed, so you can always reprocess it. Clean (silver) data is validated, deduplicated, and converted to an efficient format. Curated (gold) data is shaped for specific uses such as dashboards and models.
Sources --> RAW (bronze) --> CLEAN (silver) --> CURATED (gold) --> BI, ML, apps
as received validated, typed, business-ready
never edited deduplicated aggregatesStep 4: See the building blocks of a lake on AWS
Amazon S3 provides the storage. AWS Glue provides ingestion and transformation jobs and the Data Catalog, which records which tables exist and where. Amazon Athena queries data in place with SQL. AWS Lake Formation and IAM control access. Other clouds offer equivalents: Azure Data Lake Storage with Synapse, and Google Cloud Storage with BigQuery.
Storage Amazon S3
Catalog AWS Glue Data Catalog
Processing AWS Glue (Spark), Amazon EMR, Athena
Query Amazon Athena, Amazon Redshift Spectrum
Security IAM, AWS Lake Formation, AWS KMS
Streaming in Amazon Kinesis Data Firehose, Amazon MSKStep 5: Avoid the data swamp
A lake without ownership, naming rules, a catalog, and quality checks turns into a data swamp: nobody knows what files mean or whether to trust them. Prevent it from day one with consistent folder names, a catalog entry for every dataset, an owner for each source, documented schemas, and automated quality checks. The remaining modules build these habits in.
FAQs
- Is a data lake a replacement for a data warehouse?
- Not usually. A lake is cheap, flexible storage for all data, and a warehouse is optimized for fast, governed reporting. Lakehouse formats blur the line, but most organizations still use both for different jobs.
Summary
A data lake stores raw data cheaply and applies structure later. Organize it in raw, clean, and curated layers, use a catalog and ownership rules to avoid a data swamp, and consider a lakehouse table format for reliable SQL on files.
Related tutorials
- Databricks Module 8: SQL Warehouses, Performance, Cost, and a First Look at MLflow
- Databricks Module 7: Jobs, Orchestration, and Deploying with Git and Bundles
- Databricks Module 6: Declarative Pipelines and the Medallion Architecture
- Databricks Module 5: Ingesting Files with Auto Loader and COPY INTO
- Databricks Module 4: Unity Catalog, Volumes, and Governance
Comments
Log in or create a free account to join the discussion.