Data Lake Architecture on AWS: S3, Glue, and Athena Explained

A data lake on AWS is a centralized repository built on Amazon S3 that lets you store structured, semi-structured, and unstructured data at any scale, then query it in place using Amazon Athena without moving it into a separate database. This [architecture](https://www.draft1.ai/blog/architecture-diagram-generator-the-definitive-guide-for-cloud-engineers-in-2026) has become the default starting point for analytics workloads because it decouples storage from compute, keeps costs predictable, and integrates cleanly with the rest of the AWS ecosystem. In this article you will learn how S3, AWS Glue, and Athena fit together, where each service draws its boundaries, and how to avoid the most common design mistakes.

Whether you are migrating a legacy data warehouse, building a new analytics platform from scratch, or trying to make sense of an existing pile of S3 buckets, understanding the roles of these three services is the foundation everything else rests on.

What Is a Data Lake and Why AWS S3 Is the Default Storage Layer

Amazon S3 is the de-facto storage backbone of a data lake because it offers virtually unlimited capacity, eleven-nines durability, and a pay-per-GB pricing model that scales to zero when data is not being read. Unlike a data warehouse, which enforces a schema at write time, S3 accepts any file format — CSV, JSON, Parquet, ORC, Avro, Delta Lake, Apache Iceberg — and lets you impose structure later at query time.

A well-designed S3 data lake is typically organized into three logical zones:

• Raw (Bronze): Immutable landing zone. Data arrives exactly as the source system produced it. No transformations, no deletions.
• Curated (Silver): Cleaned, deduplicated, and lightly transformed data, usually converted to a columnar format such as Parquet or ORC.
• Aggregated (Gold): Business-ready datasets — daily summaries, dimensional models — optimized for BI tools and ad-hoc queries.

Prefix design matters enormously for performance. Partitioning by year=YYYY/month=MM/day=DD lets Athena prune entire S3 prefixes before scanning a single byte, which directly reduces query cost. S3 supports up to 5,500 GET requests per second per prefix, so spreading data across multiple prefixes also prevents throttling under heavy read workloads.

S3 storage classes add another dimension. Raw data that is rarely re-read after 30 days is a natural candidate for S3-Infrequent Access or S3 Glacier Instant Retrieval. S3 Lifecycle policies can automate the transition, keeping storage costs in check without manual intervention.

AWS Glue: The Metadata and ETL Layer

AWS Glue is a fully managed serverless service that provides two distinct capabilities: a Data Catalog that stores table metadata, and a distributed ETL engine (based on Apache Spark) that transforms data between zones. Conflating these two roles is one of the most common architectural mistakes — you can use the Glue Data Catalog without running a single Glue ETL job, and you can run Glue ETL jobs without ever touching the catalog.

The Glue Data Catalog

The Glue Data Catalog is a Hive-compatible metastore. It stores database and table definitions — column names, data types, partition keys, S3 locations, and SerDe properties — that Athena, Amazon EMR, and Amazon Redshift Spectrum all read from the same source of truth. This single catalog eliminates the "which schema is correct?" problem that plagues teams who maintain separate metadata stores per tool.

Glue Crawlers automate catalog population. A crawler points at an S3 prefix, samples the files, infers the schema, and writes or updates table definitions in the catalog. Crawlers are convenient for initial setup but can be unreliable for production pipelines where schema changes need to be reviewed before they propagate. Many teams run crawlers on a schedule in development and switch to Glue Data Catalog API calls (or Terraform/CDK definitions) in production for tighter control.

Glue ETL Jobs

Glue ETL jobs run on a managed Spark cluster. You write transformation logic in Python or Scala using either the native Spark API or Glue's higher-level DynamicFrame abstraction, which handles schema inconsistencies that would cause a standard Spark DataFrame to fail. As of 2025, Glue 4.0 is the current major version, supporting Spark 3.3 and Python 3.10.

Key configuration decisions for Glue ETL:

• Worker type: G.1X (1 DPU, 4 vCPU, 16 GB RAM) for most workloads; G.2X for memory-intensive joins; G.025X for lightweight streaming jobs.
• Job bookmarks: Enable these to track which S3 objects have already been processed, preventing reprocessing on incremental runs.
• Glue Streaming vs. batch: Glue Streaming uses micro-batches (default 100-second windows) and is suitable for near-real-time ingestion from Kinesis or Kafka. For sub-second latency, consider Apache Flink on Amazon Managed Service for Apache Flink instead.

Amazon Athena: Serverless SQL on S3

Amazon Athena is a serverless interactive query service that executes standard SQL against data stored in S3, charging $5 per terabyte of data scanned (as of 2026). There is no cluster to provision, no idle cost, and queries can start returning results within seconds. Athena is built on Presto/Trino and supports ANSI SQL, window functions, geospatial queries, and user-defined functions written in Java or Python.

Reducing Athena Query Cost

Because billing is based on bytes scanned, format and partitioning choices have a direct dollar impact:

1. Use columnar formats. Parquet or ORC files allow Athena to read only the columns referenced in a query. A query selecting 3 columns from a 100-column Parquet table may scan 97% less data than the equivalent CSV query.
2. Partition your tables. Declare partition columns in the Glue catalog and use MSCK REPAIR TABLE or Glue partition indexing to register new partitions. Athena's partition pruning can reduce scanned data by orders of magnitude.
3. Compress your files. Snappy compression (the Parquet default) typically reduces file size by 60–70% compared to uncompressed CSV.
4. Right-size your files. Aim for 128 MB–1 GB per file. Thousands of tiny files cause excessive S3 LIST calls and slow query planning.

Athena Workgroups and Access Control

Workgroups let you isolate query history, enforce per-query data scan limits, and route costs to specific teams or projects. A workgroup can cap any single query at, say, 10 GB of scanned data, preventing runaway queries from generating unexpected bills. Combined with IAM policies and AWS Lake Formation column-level and row-level access controls, workgroups give you a complete governance layer without a separate security tool.

How S3, Glue, and Athena Work Together: End-to-End Flow

A typical pipeline looks like this:

1. Ingest: Raw data lands in s3://my-lake/raw/ via Kinesis Firehose, AWS DataSync, or a custom Lambda function.
2. Catalog: A Glue Crawler (or an explicit CREATE TABLE DDL in Athena) registers the raw table in the Glue Data Catalog.
3. Transform: A Glue ETL job reads the raw DynamicFrame, applies cleaning logic, converts to Parquet, and writes to s3://my-lake/curated/.
4. Re-catalog: The ETL job calls the Glue catalog API to add new partitions, or Athena's ALTER TABLE ADD PARTITION statement is executed.
5. Query: Analysts run SQL in the Athena console, a BI tool via the Athena JDBC/ODBC driver, or programmatically via the AWS SDK.
6. Govern: AWS Lake Formation enforces column masking and row filters so different teams see only the data they are authorized to access.

Comparing Storage Format Options for Athena Queries

Choosing the right file format is one of the highest-leverage decisions in a data lake design.

Apache Iceberg deserves special mention. Athena has supported Iceberg tables natively since 2022, and by 2026 it is the recommended format for any dataset that requires UPDATE, DELETE, or MERGE operations — use cases that plain Parquet cannot handle without rewriting entire partitions.

Key Takeaways

• S3 is storage, Glue is metadata and transformation, Athena is query — keep these roles distinct in your architecture diagrams and cost models.
• Partition your S3 data by the columns most commonly used in WHERE clauses; this is the single highest-impact optimization for Athena query cost.
• Convert raw data to Parquet or ORC in the curated zone; columnar formats can reduce Athena scan costs by 60–90% compared to CSV.
• Use the Glue Data Catalog as a single metastore shared across Athena, EMR, and Redshift Spectrum to avoid schema drift between tools.
• Enable Glue job bookmarks on incremental ETL jobs to prevent duplicate processing without writing custom state management code.
• Use Apache Iceberg tables in Athena when your dataset requires row-level updates or deletes; plain Parquet is append-only without full partition rewrites.
• Apply AWS Lake Formation row and column-level security on top of Athena workgroups to enforce data governance without duplicating data into separate stores.

Frequently Asked Questions

What is the difference between a data lake and a data warehouse on AWS?

A data lake stores raw data in open file formats on S3 and applies schema at query time, while a data warehouse like Amazon Redshift stores structured data in a proprietary columnar format and enforces schema at load time. Data lakes are cheaper per GB and more flexible, but data warehouses typically deliver faster, more consistent query performance for complex analytical workloads. Many production architectures use both: a lake for raw storage and exploration, a warehouse for governed reporting.

Do I need AWS Glue to use Athena?

No — Athena can use any Hive-compatible metastore, including the Glue Data Catalog, an external Apache Hive metastore, or tables defined directly with DDL statements in Athena itself. However, the Glue Data Catalog is the default and most convenient option because it is fully managed, shared across services, and free for the first million objects stored.

How much does an Athena query actually cost?

Athena charges $5 per TB of data scanned, rounded up to the nearest 10 MB with a 10 MB minimum per query. A query that scans 500 MB of Parquet data costs roughly $0.0025. Queries against DDL statements, failed queries, and cancelled queries before data is scanned are not charged. Compression and columnar formats are the primary levers for reducing this cost.

Can Athena handle UPDATE and DELETE operations?

Yes, but only on Apache Iceberg tables. Standard Athena tables backed by Parquet or ORC are effectively immutable — you can append new files but cannot modify existing rows without rewriting partitions. Iceberg's ACID transaction support, available natively in Athena, enables UPDATE, DELETE, and MERGE INTO statements on S3-backed data.

What is the maximum size of data Athena can query?

Athena has no documented upper limit on the total size of data it can query. In practice, queries against petabyte-scale datasets work correctly as long as partitioning and file sizing are well-designed. Individual query results are limited to 1,000 rows in the console but unlimited when written to an S3 output location.

How do Glue Crawlers handle schema changes?

By default, a Glue Crawler will add new columns to an existing table definition but will not remove columns that disappear from the source data. If a crawler detects a fundamentally incompatible schema change (for example, a column changing from string to integer), it creates a new table rather than overwriting the existing one. For production pipelines, many teams disable automatic crawler updates and manage schema evolution explicitly through the Glue API or infrastructure-as-code.

Is AWS Lake Formation required for a data lake on AWS?

Lake Formation is optional but strongly recommended for any multi-team or production environment. Without it, access control relies entirely on S3 bucket policies and IAM, which operate at the file level and cannot enforce column masking or row-level filters. Lake Formation adds a permissions layer on top of the Glue catalog that lets you grant table-, column-, and row-level access to IAM principals without duplicating or moving data.

Draw this in seconds with draft1. Describe your architecture in plain English and draft1 generates an editable AWS/cloud diagram plus documentation — no dragging boxes around. Try it free.