Can I convert existing Parquet to Iceberg?

Yes. Engines offer ways to create an Iceberg table from existing data, either by rewriting the files or by registering existing files. Test the method on a copy first and follow the documentation for your engine.

Building a Data Lake Module 6: Open Table Formats with Iceberg, Delta Lake, and Hudi

Introduction

Plain Parquet files are simple, but they cannot safely update or delete rows, and concurrent writers can leave readers with half-written data. Open table formats add a metadata layer that gives lake files database-like behavior. This module compares the three main formats and creates an Iceberg table in Athena.

Prerequisites

Module 5 completed

Step-by-Step Guide

Step 1: Learn what a table format adds

A table format tracks which data files belong to a table in metadata. That enables ACID transactions (readers see a consistent snapshot), row-level updates, deletes, and merges, schema evolution, time travel to earlier versions, and hidden partitioning in some formats. It also lets different engines work with the same table safely.

Plain Parquet folder     files only, no transactions, rewrite everything to change a row
Iceberg / Delta / Hudi  files + metadata = ACID, MERGE, schema evolution, time travel

Step 2: Compare Iceberg, Delta Lake, and Hudi

Apache Iceberg is vendor-neutral with broad support across engines, including Athena, Glue, Redshift, Snowflake, Spark, Trino, and Flink, and is a common default on AWS. Delta Lake is tied most closely to Databricks and Spark, with a strong ecosystem there. Apache Hudi is strong for streaming upserts and change data capture. Check each format's version and engine support before committing, because this area changes quickly.

Format   Strengths                              Often chosen when
Iceberg  engine-neutral, partition evolution    multi-engine lake on AWS or elsewhere
Delta    tight Databricks/Spark integration     your platform is Databricks
Hudi     record-level upserts, CDC pipelines    heavy streaming updates

Step 3: Create an Iceberg table in Athena

Athena can create and manage Iceberg tables. Set the table type property and a location in S3, then insert and query as usual. Athena supports row-level operations such as UPDATE, DELETE, and MERGE on Iceberg tables.

CREATE TABLE sales_lake.customers_iceberg (
  customer_id bigint,
  name string,
  city string,
  updated_at timestamp
)
LOCATION 's3://acme-datalake-curated-dev/sales/customers_iceberg/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

INSERT INTO sales_lake.customers_iceberg VALUES
  (1, 'Asha', 'Pune', current_timestamp),
  (2, 'Ravi', 'Delhi', current_timestamp);

Step 4: Update data with MERGE and travel back in time

MERGE applies inserts and updates from a source in one transaction, which is how change data from a source system lands in a lake table. Time travel reads the table as it was at an earlier moment, helping you audit changes or recover from a bad load.

MERGE INTO sales_lake.customers_iceberg t
USING sales_lake.customers_updates s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET city = s.city, updated_at = s.updated_at
WHEN NOT MATCHED THEN INSERT (customer_id, name, city, updated_at)
  VALUES (s.customer_id, s.name, s.city, s.updated_at);

-- read the table as of an earlier time
SELECT * FROM sales_lake.customers_iceberg
FOR TIMESTAMP AS OF (current_timestamp - interval '1' day);

Step 5: Maintain tables: compaction and snapshot cleanup

Every write adds files and snapshots, so tables need regular maintenance: compact small files into larger ones, and expire old snapshots and remove orphan files when you no longer need history. Athena provides OPTIMIZE and VACUUM statements for Iceberg tables. Amazon S3 Tables is a managed option that handles Iceberg compaction automatically. Check the current documentation for details and limits.

OPTIMIZE sales_lake.customers_iceberg REWRITE DATA USING BIN_PACK;
VACUUM sales_lake.customers_iceberg;

FAQs

Can I convert existing Parquet to Iceberg?: Yes. Engines offer ways to create an Iceberg table from existing data, either by rewriting the files or by registering existing files. Test the method on a copy first and follow the documentation for your engine.

Summary

Open table formats add transactions, updates, deletes, schema evolution, and time travel to lake files. Iceberg is a common engine-neutral default on AWS, Delta fits Databricks, and Hudi suits streaming upserts. Maintain tables with compaction and snapshot expiry.