Blog >

Data Lake Architecture on AWS: Building with S3, Glue, and Athena

Posted by Hadi @draft1 | November 3, 2026

Data Lake Architecture on AWS: Building with S3, Glue, and Athena

A data lake on AWS is a centralized storage layer—usually Amazon S3—paired with a shared metadata catalog and query engines like AWS Glue and Amazon Athena that let you analyze raw and processed data without moving it. This pattern decouples storage from compute, so you pay for data once and query it with whichever engine fits the job, from ad hoc SQL to Spark ETL to machine learning feature extraction.

This article walks through a practical reference architecture: how S3 buckets and prefixes should be organized, how Glue Crawlers and the Glue Data Catalog work, how Athena executes queries against that catalog, and where things commonly go wrong—partitioning mistakes, small-file problems, cost surprises, and schema drift. It's written for engineers who are past the "what is a data lake" stage and want to actually build one.

What a Modern AWS Data Lake Actually Consists Of

At minimum, an AWS data lake is three things working together: S3 for durable, cheap object storage, the Glue Data Catalog as a central metastore, and a query layer (Athena, Redshift Spectrum, EMR, or all three) that reads catalog metadata to know how to interpret files in S3.

S3 provides 11 nines of durability (per AWS's published design target) and effectively unlimited scale, at a fraction of the cost of block or file storage. But S3 alone is just a bucket of objects — it has no concept of tables, columns, or schemas. That's where the Glue Data Catalog comes in: it's a persistent, Hive-compatible metastore that stores table definitions (location, file format, partition keys, column types) so that multiple compute engines can share one source of truth. Athena, Redshift Spectrum, EMR (via Hive/Spark), and even third-party tools like Trino can all point at the same Glue Catalog and see the same tables.

The typical data flow looks like this:

  1. Raw data lands in S3 (via Kinesis Data Firehose, DMS, batch uploads, or application writes).
  2. Glue Crawlers or Glue ETL jobs infer or enforce schema and register tables in the Data Catalog.
  3. Glue ETL jobs (Spark or Python Shell) transform raw data into curated, partitioned, columnar formats.
  4. Athena, QuickSight, SageMaker, or Redshift Spectrum query the curated layer.

Designing the S3 Layer: Zones, Prefixes, and File Formats

A well-organized S3 layer separates data by processing stage (raw, curated, consumption) and encodes partitioning in the prefix structure so query engines can skip irrelevant data. Getting this wrong is the single biggest cause of slow, expensive Athena queries.

Most teams use a three-zone (sometimes called medallion) layout:

  • Raw/Bronze zone — immutable, exactly-as-ingested data (JSON, CSV, log files). Treated as a permanent record.
  • Curated/Silver zone — cleaned, deduplicated, type-enforced data, usually converted to Parquet or ORC and compressed with Snappy or Zstandard.
  • Consumption/Gold zone — aggregated, business-ready tables optimized for specific dashboards or ML features.

A bucket naming convention example:

s3://acme-datalake-raw/orders/ingest_date=2026-01-15/
s3://acme-datalake-curated/orders/year=2026/month=01/day=15/
s3://acme-datalake-gold/orders_daily_summary/year=2026/month=01/

Partitioning by date (or another high-cardinality, frequently-filtered column like region or tenant_id) is what lets Athena perform partition pruning — skipping S3 prefixes that can't match the query's WHERE clause instead of scanning the whole table. For time-series data, a year/month/day (or year/month/day/hour) hierarchy is standard; avoid partitioning on columns with extremely high cardinality (like user_id) because it creates millions of tiny partitions and overwhelms the Glue Catalog and S3 LIST operations.

File format choice matters just as much as partitioning:

  • Parquet is columnar, splittable, and supports predicate pushdown — Athena only reads the columns and row groups it needs. This is the default recommendation for analytical workloads.
  • ORC is similar to Parquet and preferred in some Hive/EMR-heavy shops; Athena supports both equally well.
  • CSV/JSON are fine for raw ingestion but expensive to query at scale because Athena must scan entire rows and files.

A practical rule: convert raw CSV/JSON to Parquet as early as possible in the pipeline, ideally during the first Glue ETL hop, not at query time.

The Glue Data Catalog and Crawlers

The Glue Data Catalog is a managed, Hive Metastore-compatible service that stores table and partition metadata; Glue Crawlers populate it automatically by scanning S3 data and inferring schema and partition structure.

A crawler connects to a data store (S3 path, JDBC source, DynamoDB table), samples files, infers columns and types, detects partition keys from the folder structure, and writes or updates table definitions in the catalog. Crawlers are convenient for onboarding new, evolving datasets, but they have real trade-offs:

  • Schema drift handling: a crawler can be configured to add new columns, mark tables as needing manual review, or ignore changes — picking the wrong policy silently breaks downstream queries.
  • Cost and latency: each crawler run is billed per DPU-hour (AWS Glue crawler pricing), and running one every few minutes on a large bucket is both slow and wasteful.
  • Partition explosion: crawling a bucket with poorly designed partitioning can take a very long time and create an unwieldy number of partition objects in the catalog.

In production pipelines, many teams crawl once to establish structure, then switch to explicit partition management — using ALTER TABLE ... ADD PARTITION, AWS Glue's create_dynamic_frame with the catalog API, or Glue ETL jobs that call glueContext.create_partitioning — rather than re-crawling on every ingest. This avoids the DPU cost of constant crawling and gives you deterministic control over schema evolution.

Glue ETL Jobs: Spark, Python Shell, and Serverless DPUs

AWS Glue ETL is a serverless Spark (or Python Shell) environment billed per DPU-hour, used to transform raw data into curated, partitioned, columnar tables registered in the same Data Catalog that Athena queries.

A Glue job typically:

  1. Reads a DynamicFrame from a Glue Catalog table (source = raw zone).
  2. Applies transformations — type casting, deduplication, flattening nested JSON, joining reference data.
  3. Writes out as Parquet, partitioned by date or another key, to the curated zone.
  4. Updates the Data Catalog (via enableUpdateCatalog or a follow-up crawler) so Athena immediately sees the new partitions.

Glue offers two execution modes worth distinguishing:

  • Glue Spark jobs — for large-scale, distributed transforms (joins across large tables, complex aggregations). Billed per DPU-hour with a minimum duration; Glue 4.0/5.0 runtimes support Spark 3.x and improved Parquet writers.
  • Glue Python Shell jobs — for lightweight, single-node scripts (small file manipulation, calling APIs, pandas-based transforms) at a fraction of the cost of spinning up a Spark cluster.

For orchestration, Glue Workflows or Step Functions chain crawlers and jobs together (e.g., crawl → job A → job B → crawl again), with EventBridge triggering runs on S3 object-created events for near-real-time pipelines.

Querying with Athena

Amazon Athena is a serverless, interactive query engine based on Trino/Presto that queries data directly in S3 using table definitions from the Glue Data Catalog, billed by the amount of data scanned per query.

Key operational facts:

  • Pricing (per AWS's published Athena rate) is $5 per terabyte of data scanned, rounded up, with a minimum of 10 MB per query — which is exactly why partitioning and columnar formats matter: they directly reduce the bytes scanned and therefore the bill.
  • Athena has a default 30-minute query timeout per query (adjustable via workgroup settings up to a service limit) and a result size limit enforced through S3 output location configuration.
  • Workgroups let you isolate teams, set per-query or per-workgroup data-scan limits, and separate billing/cost tracking — essential in multi-tenant data lakes.
  • Athena supports CTAS (CREATE TABLE AS SELECT) and INSERT INTO for materializing query results as new partitioned Parquet tables, which is a common pattern for building the curated or gold layer without standing up a Spark job.
  • Federated queries via Athena Data Source Connectors (Lambda-based) let you join S3 data with RDS, DynamoDB, or other sources in a single query, at the cost of additional Lambda invocation latency.

A simple example of enforcing partition pruning in a query:

SELECT customer_id, SUM(order_total) AS total_spent
FROM curated.orders
WHERE year = '2026' AND month = '01' AND day BETWEEN '01' AND '15'
GROUP BY customer_id;

Because year, month, and day are partition keys in the Glue table definition, Athena only scans the matching S3 prefixes rather than the entire orders table — the difference between scanning a few GB and scanning terabytes.

Where Each Engine Fits: A Comparison

No single tool does everything well; choosing between Athena, Glue ETL, Redshift Spectrum, and EMR is about workload shape, not which is "best."

Tool Best for Pricing model Key limitation
Athena Ad hoc SQL, BI dashboards, CTAS table builds Per TB scanned ($5/TB) 30-min default query timeout
Glue ETL (Spark) Large-scale batch transforms, complex joins Per DPU-hour Cold-start latency per job
Redshift Spectrum Joining S3 data with a Redshift warehouse Redshift cluster/Serverless cost + per-TB scan Requires an active Redshift cluster
EMR Custom Spark/Hive/Presto workloads, ML pipelines Per-instance-hour (EC2 + EMR fee) More operational overhead

Cost and Performance Optimization

The majority of data lake cost overruns on AWS come from scanning too much data, not from storage itself — S3 storage is cheap; inefficient queries and over-crawling are not.

Concrete levers, roughly in order of impact:

  1. Convert to Parquet/ORC and compress (Snappy/Zstd). This alone commonly cuts Athena scan costs by 80–90% compared to raw JSON/CSV, since columnar formats let Athena read only needed columns and compression reduces bytes read.
  2. Partition correctly on columns actually used in WHERE clauses — don't partition on something no query filters by, and don't over-partition on high-cardinality fields.
  3. Compact small files. S3 and Athena perform poorly with millions of tiny objects (a classic symptom of frequent small Kinesis Firehose or streaming writes); use Glue ETL or Firehose buffering (e.g., 128 MB buffer size) to produce fewer, larger files — a good target is 128 MB–1 GB per file.
  4. Use S3 Lifecycle policies to transition cold raw data to S3 Glacier Instant Retrieval or Glacier Deep Archive, since raw zones often need to be retained for compliance but are rarely queried after curation.
  5. Set Athena workgroup data-scan limits to catch runaway queries before they generate a large bill.
  6. Use CTAS with partitioning to pre-aggregate hot queries into gold tables rather than re-scanning raw curated data repeatedly.

Security, Governance, and Lake Formation

AWS Lake Formation sits on top of S3 and the Glue Catalog to provide fine-grained, column- and row-level access control, centralizing permissions that would otherwise require a patchwork of S3 bucket policies and IAM conditions.

Without Lake Formation, access control in a data lake is typically enforced at the S3 bucket/prefix level via IAM policies — workable for coarse separation (e.g., "finance team can read s3://lake/finance/*") but clumsy for column-level restrictions (e.g., masking a ssn column from analysts while allowing access to the rest of a table). Lake Formation grants permissions at the database, table, column, and even row level, and those permissions are enforced consistently whether the query comes from Athena, Redshift Spectrum, or EMR — because all of them check against the same Lake Formation permission model layered on the Glue Catalog.

Other governance essentials:

  • Encryption: S3 server-side encryption (SSE-S3 or SSE-KMS) for data at rest; Athena query results should also be encrypted in their S3 output location.
  • Audit: AWS CloudTrail logs Glue Catalog API calls and Athena query executions; AWS Glue Data Quality (built on Deequ) can enforce rules before data is promoted from raw to curated.
  • Tagging and cost allocation: tag S3 buckets, Glue jobs, and Athena workgroups so Cost Explorer can attribute spend per team or pipeline.

Key Takeaways

  • A data lake on AWS separates storage (S3) from compute (Athena, Glue, EMR, Redshift Spectrum), with the Glue Data Catalog acting as the shared metadata layer all engines read from.
  • Organize S3 into raw, curated, and consumption zones, and partition by columns actually used in query filters — this is the single biggest lever for Athena performance and cost.
  • Convert data to Parquet or ORC as early as possible; it typically cuts Athena scan costs dramatically compared to raw JSON/CSV.
  • Glue Crawlers are convenient for schema discovery but cost DPU-hours and can mishandle schema drift — many production pipelines manage partitions explicitly instead of re-crawling constantly.
  • Athena bills $5 per terabyte scanned (per AWS's published rate) with a 10 MB minimum per query, so compaction and partition pruning directly reduce your bill.
  • Use AWS Lake Formation when you need column- or row-level access control that's consistent across Athena, Redshift Spectrum, and EMR.
  • Choose your compute engine by workload shape: Athena for ad hoc SQL and BI, Glue Spark for heavy ETL, Redshift Spectrum when you already run Redshift, EMR for custom Spark/ML pipelines.

Frequently Asked Questions

What's the difference between a data lake and a data warehouse on AWS?

A data lake (S3 + Glue + Athena) stores raw and semi-structured data cheaply and lets multiple engines query it on demand, while a data warehouse like Amazon Redshift stores highly structured data in its own managed storage optimized for complex joins and fast repeated BI queries. Many architectures use both — a "lakehouse" pattern — where Redshift Spectrum or Redshift's zero-ETL integrations query S3 data directly alongside warehouse tables.

Do I need Glue Crawlers if I already know my schema?

No — if your schema and partitions are stable and known in advance, you can create Glue Catalog tables directly via the AWS SDK, Glue API, or Terraform/CloudFormation, and manage partitions explicitly with ALTER TABLE ADD PARTITION or Athena's MSCK REPAIR TABLE. This avoids crawler DPU costs and gives you more predictable schema evolution behavior.

How does Athena pricing actually work?

Athena charges $5 per terabyte of data scanned (as published in AWS's Athena pricing), rounded up per query with a 10 MB minimum, regardless of how long the query takes. This is why reducing bytes scanned — via Parquet, compression, and partition pruning — has a direct and often dramatic effect on cost, unlike provisioned compute pricing.

Can Athena handle real-time or streaming data?

Not natively — Athena queries data already sitting in S3, so "near-real-time" means minimizing the delay between data arriving and being queryable. A common pattern uses Kinesis Data Firehose to buffer and write data to S3 every 60–900 seconds, after which it's immediately queryable by Athena once the Glue Catalog reflects the new partition.

What causes the "small files problem" in a data lake, and how do you fix it?

It happens when streaming ingestion or poorly tuned ETL jobs write many tiny objects (a few KB each) instead of fewer large ones, which increases S3 LIST overhead and slows query planning in Athena, Glue, and EMR. Fix it by tuning Firehose buffer sizes, running periodic Glue compaction jobs, or using CTAS in Athena to rewrite small-file partitions into fewer, larger Parquet files.

Is Lake Formation required to build a data lake on AWS?

No, Lake Formation is optional — you can build a functional data lake with just S3, Glue, and Athena using IAM and S3 bucket policies for access control. Lake Formation becomes valuable once you need fine-grained column- or row-level permissions, or centralized governance across multiple query engines and teams.

Should I use Parquet or stick with CSV for a new data lake?

Use Parquet (or ORC) for anything you'll query repeatedly with Athena, Redshift Spectrum, or Spark — its columnar layout and compression typically reduce scan costs and query time substantially compared to CSV. CSV or JSON is acceptable only for the initial raw-ingestion zone, where data is written once and converted to a columnar format before serious analytical use.


Draw this in seconds with draft1. Describe your architecture in plain English and draft1 generates an editable AWS/cloud diagram plus documentation — no dragging boxes around. Try it free.

Draw this in ~20 seconds

Describe your own version of this architecture and draft1 generates an editable draw.io diagram — boxes, arrows, labels, the lot.

Generate this diagram free ➔

Free demo — no signup. Then 3 free diagrams with an account, no card.