Why did my crawler create several tables instead of one?

Crawlers group files into a table when their schemas are similar and they share a path structure. Files with different schemas in the same folder, or mixed formats, can become separate tables. Keep one dataset per folder with a consistent schema.

AWS Glue Module 2: The Data Catalog and Crawlers

Introduction

The AWS Glue Data Catalog is the shared map of your data: which tables exist, their columns and types, and where the files live. A crawler builds that map automatically by reading a sample of your files. This module catalogs the sample orders file and shows how to query it.

Prerequisites

Module 1 completed

Step-by-Step Guide

Step 1: Create a catalog database

A database in the catalog is just a namespace that groups tables. It does not hold data. Create one for the course.

aws glue create-database --database-input '{"Name": "glue_lab"}'
aws glue get-databases --query "DatabaseList[].Name"

Step 2: Create and run a crawler

A crawler needs a role, a target (an S3 path here), and a target database. It samples files, infers the schema and format, and creates or updates tables. Add a table prefix so crawler-created tables are easy to recognize.

aws glue create-crawler --name orders-raw-crawler \
  --role GlueLabRole \
  --database-name glue_lab \
  --table-prefix raw_ \
  --targets '{"S3Targets":[{"Path":"s3://my-glue-lab-123456/raw/orders/"}]}'

aws glue start-crawler --name orders-raw-crawler
aws glue get-crawler --name orders-raw-crawler --query "Crawler.State"

Step 3: Inspect the table the crawler created

When the crawler finishes, look at the table it produced: the column names and types it inferred, the data format, and the S3 location. Crawlers infer types from a sample, so check them. For example a column of numbers with a few text values may become a string.

aws glue get-table --database-name glue_lab --name raw_orders \
  --query "Table.StorageDescriptor.Columns"

Step 4: Query the cataloged table with Athena

Because Athena reads the Glue Data Catalog, the new table is available immediately. Open the Athena query editor, choose the glue_lab database, and run a query. Set an S3 query results location the first time you use Athena.

SELECT status, count(*) AS orders, sum(amount) AS revenue
FROM glue_lab.raw_orders
GROUP BY status;

Step 5: Control crawler behavior and cost

Decide what the crawler does when it finds schema changes or deleted data, using its schema change policy. Run crawlers on a schedule only as often as data changes, because each run is billed. For large, stable datasets many teams skip crawlers and define tables explicitly with SQL (as in the data lake course) or use partition projection. Crawlers are convenient for discovery, not always the best production choice.

aws glue update-crawler --name orders-raw-crawler \
  --schema-change-policy UpdateBehavior=LOG,DeleteBehavior=LOG

# run on a schedule (cron): every day at 02:00 UTC
aws glue update-crawler --name orders-raw-crawler --schedule "cron(0 2 * * ? *)"

FAQs

Why did my crawler create several tables instead of one?: Crawlers group files into a table when their schemas are similar and they share a path structure. Files with different schemas in the same folder, or mixed formats, can become separate tables. Keep one dataset per folder with a consistent schema.

Summary

The Data Catalog stores table definitions used by Glue, Athena, EMR, and Redshift. A crawler samples files to create tables, you should check inferred types, schedule crawlers sensibly, and consider explicit table definitions for stable production datasets.