Building a Data Lake Module 4: Ingesting Data into the Lake

Step 1: Match the ingestion method to the source

For files, copy them into the raw zone with the AWS CLI, an SDK, AWS Transfer Family for SFTP, or S3 replication. For relational databases, AWS Database Migration Service can do a full load and then continuous change data capture (CDC) into S3. For SaaS applications, Amazon AppFlow offers managed connectors. For events and logs, Amazon Kinesis Data Firehose or Amazon MSK deliver streams to S3 in batches.

Source type            Typical AWS option
Files / SFTP           S3 upload, AWS Transfer Family
Operational database   AWS DMS (full load + CDC), or Glue JDBC extracts
SaaS apps              Amazon AppFlow
Application events     Kinesis Data Streams / Data Firehose, Amazon MSK
Logs                   CloudWatch, Firehose -> S3

Step 2: Land data with a consistent, immutable path

Write each load to a new path that includes the source, dataset, and ingestion date, and never overwrite what is already there. Immutable raw data makes bugs recoverable and gives you an audit trail. Add the load time or batch id to file names.

import boto3
from datetime import datetime, timezone

s3 = boto3.client("s3")
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
batch = datetime.now(timezone.utc).strftime("%H%M%S")

s3.upload_file(
    "orders.json",
    "acme-datalake-raw-dev",
    "sales/orders/ingest_date=" + today + "/orders-" + batch + ".json",
)

Step 3: Stream events with Kinesis Data Firehose

Firehose is a managed delivery stream: producers send records, and Firehose buffers them (by size or time) and writes files to S3, optionally converting to Parquet and partitioning by time. Create the stream and its IAM role in the console or with infrastructure as code, then send records from your application.

import boto3, json

firehose = boto3.client("firehose")
event = {"user_id": 42, "action": "add_to_cart", "sku": "A-100"}

firehose.put_record(
    DeliveryStreamName="clickstream-to-s3",
    Record={"Data": (json.dumps(event) + "\n").encode("utf-8")},
)

Step 4: Make loads idempotent and trackable

Pipelines fail and are rerun, so a rerun must not create duplicates. Write each batch to a deterministic path so a retry overwrites the same files, record what has been loaded (a manifest or a control table), and include a unique id in each record so duplicates can be removed later. Capture metadata such as the source system, load time, and file name alongside the data.

Step 5: Validate on the way in

Run light checks as data lands: the file is not empty, the expected columns exist, and row counts are plausible compared with the source. Quarantine bad files in a separate prefix instead of deleting them, and alert the owner. Deeper cleaning belongs in the next layer, not in the raw zone.

s3://acme-datalake-raw-dev/sales/orders/ingest_date=2026-09-16/      good files
s3://acme-datalake-raw-dev/_quarantine/sales/orders/2026-09-16/       rejected files