Does S3 have real folders?

No. S3 stores objects with keys, and the slashes in a key only look like folders in the console. This matters because renaming a folder means copying every object, so choose names carefully up front.

Building a Data Lake Module 2: Storage Foundations on Amazon S3

Introduction

Amazon S3 is the storage layer of most AWS data lakes. Decisions you make now about buckets, folder names, and security are hard to change later when you have millions of files. This module sets up the storage with the AWS CLI. Use a sandbox AWS account, and check the current S3 documentation for the latest options and prices.

Prerequisites

Module 1 completed
An AWS account and the AWS CLI configured with credentials

Step-by-Step Guide

Step 1: Choose bucket and prefix names

A bucket name must be globally unique. Many teams use one bucket per zone or per environment, or one bucket with a prefix per zone. Use lowercase, hyphens, and a clear pattern. Inside the bucket, use prefixes (the folder-like path) by zone, source, and dataset.

s3://acme-datalake-raw-dev/
    sales/orders/ingest_date=2026-09-14/orders-001.json
s3://acme-datalake-clean-dev/
    sales/orders/order_date=2026-09-14/part-0001.parquet
s3://acme-datalake-curated-dev/
    sales/daily_revenue/...

Step 2: Create the buckets

Create one bucket per zone. Outside the us-east-1 region the create command needs a location constraint. Replace the example names with your own unique names and your region.

aws s3api create-bucket --bucket acme-datalake-raw-dev --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1
aws s3api create-bucket --bucket acme-datalake-clean-dev --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1
aws s3api create-bucket --bucket acme-datalake-curated-dev --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1

Step 3: Block public access and enable encryption

Data lake buckets should never be public. Turn on S3 Block Public Access, and make sure data is encrypted at rest. New buckets encrypt objects by default with S3-managed keys; use AWS KMS keys when you need audit trails or tighter key control.

aws s3api put-public-access-block --bucket acme-datalake-raw-dev \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

aws s3api get-bucket-encryption --bucket acme-datalake-raw-dev

Step 4: Turn on versioning for the raw zone

Versioning keeps previous versions of objects, which protects against accidental overwrites and deletes. It is most valuable for the raw zone, where data should never be lost. Remember that old versions also cost storage, so pair it with a lifecycle rule.

aws s3api put-bucket-versioning --bucket acme-datalake-raw-dev \
  --versioning-configuration Status=Enabled

Step 5: Add lifecycle rules to control cost

Lifecycle rules move older data to cheaper storage classes and delete what you no longer need, including old versions and incomplete uploads. Save the rule as a JSON file and apply it. Check the current storage classes and minimum storage durations before choosing the numbers.

# lifecycle.json
{
  "Rules": [
    {
      "ID": "tier-and-expire-raw",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "Transitions": [ { "Days": 90, "StorageClass": "STANDARD_IA" } ],
      "NoncurrentVersionExpiration": { "NoncurrentDays": 60 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration --bucket acme-datalake-raw-dev \
  --lifecycle-configuration file://lifecycle.json

Step 6: Upload a first file and list it

Upload a small sample file into the raw zone using the partition-style path, then list it. When you are done experimenting, delete the files and buckets you do not need so you are not billed for storage.

echo '{"order_id": 1, "amount": 120.5}' > orders-001.json
aws s3 cp orders-001.json s3://acme-datalake-raw-dev/sales/orders/ingest_date=2026-09-14/
aws s3 ls s3://acme-datalake-raw-dev/sales/orders/ --recursive

FAQs

Does S3 have real folders?: No. S3 stores objects with keys, and the slashes in a key only look like folders in the console. This matters because renaming a folder means copying every object, so choose names carefully up front.

Summary

Plan clear bucket and prefix names, create one bucket per zone, block public access, confirm encryption, enable versioning for raw data, add lifecycle rules, and clean up sandbox resources.