A data lake on AWS typically combines Amazon S3 for storage, AWS Glue for cataloging and ETL, and Amazon Athena for SQL-based querying, all without managing servers. This pattern has become the default starting point for teams who need to centralize structured, semi-structured, and unstructured data without committing to the cost or rigidity of a traditional data warehouse.
This article walks through how S3, Glue, and Athena fit together, where each component earns its keep, and where the architecture breaks down. You'll see a concrete example pipeline, a comparison of query engines you might swap in, and the operational details — partitioning, file formats, cost drivers — that determine whether this stack performs well or quietly burns money.
What "Data Lake Architecture" Means on AWS
A data lake architecture separates storage from compute, letting you store raw data cheaply in Amazon S3 and apply schema and compute only when you query it — the "schema-on-read" model, as opposed to the "schema-on-write" model used by relational warehouses. On AWS, this is implemented with three core services plus a handful of supporting ones.
- Amazon S3 — durable, nearly unlimited object storage; the lake itself.
- AWS Glue Data Catalog — a persistent metadata store (Hive Metastore-compatible) that maps S3 paths to table definitions, partitions, and schemas.
- AWS Glue crawlers and ETL jobs — discover schema automatically and transform raw data into query-optimized formats.
- Amazon Athena — a serverless, distributed SQL engine built on Trino/Presto that queries data directly in S3 using the Glue Catalog for metadata.
Supporting services often layered in: AWS Lake Formation for fine-grained permissions, Amazon EMR for heavier Spark workloads, Amazon QuickSight for BI on top of Athena, and AWS Step Functions for orchestrating multi-stage ETL.
The appeal is that each layer is independently scalable and billed separately — you pay for S3 storage, Glue job runtime (DPU-hours), and Athena per-query data scanned, rather than for a fixed cluster running 24/7.
Storage Layer: Structuring Data in S3
The S3 layer's job is to organize raw and processed data so that downstream query engines can skip unnecessary data, not just to "hold files." Layout and file format decisions here affect query cost and speed more than almost anything else in the stack.
Most production lakes use a multi-zone (medallion) layout:
- Raw/Bronze zone — immutable landing data, exactly as ingested (JSON, CSV, logs), partitioned by ingestion date.
- Processed/Silver zone — cleaned, deduplicated, type-cast data, typically converted to Apache Parquet or ORC.
- Curated/Gold zone — aggregated, business-ready tables, often partitioned by query-relevant dimensions (e.g.,
region,customer_segment).
Example S3 key structure for the processed zone:
s3://company-datalake/processed/orders/
year=2026/month=01/day=15/part-0000.snappy.parquet
year=2026/month=01/day=16/part-0000.snappy.parquet
Partitioning by year/month/day lets Athena use partition pruning — scanning only the folders relevant to a query's WHERE clause instead of the entire table. Combined with columnar formats like Parquet, which let Athena read only the columns referenced in a query, partition pruning is the single biggest lever for controlling Athena cost, since Athena bills $5 per TB scanned (US East pricing, AWS, as of 2026).
Other storage details that matter:
- File size: aim for 128 MB–1 GB per file. Too many small files (the "small file problem") forces Athena/Trino to issue excessive S3 GET requests, slowing queries and increasing per-request cost.
- Compression: Snappy is the default for Parquet in Glue because it's splittable and fast to decompress; GZIP compresses tighter but is not splittable for some formats.
- S3 Storage Classes: raw zones can move to S3 Intelligent-Tiering or S3 Glacier Instant Retrieval after a retention window using lifecycle policies, since raw data is rarely re-queried once processed.
- Table formats: for workloads needing row-level updates/deletes or ACID transactions (e.g., GDPR deletion requests, CDC ingestion), teams increasingly layer Apache Iceberg, Apache Hudi, or Delta Lake on top of S3 — all now natively supported by Athena and Glue as of recent AWS releases.
Cataloging and ETL with AWS Glue
AWS Glue provides the metadata layer that makes S3 queryable as tables, and the serverless Spark environment that transforms raw files into the optimized formats described above. Without a catalog, Athena has no concept of "tables" — it would need to be pointed at exact file paths and manually told the schema for every query.
A typical Glue workflow:
- Crawlers scan an S3 prefix, infer schema (column names, types), detect partitions from folder structure, and register a table definition in the Glue Data Catalog.
- Glue ETL jobs (PySpark or Scala, running on Glue 4.0/5.0 which is based on Spark 3.x) read raw data, apply transformations — deduplication, type coercion, joins, flattening nested JSON — and write Parquet output back to S3.
- Glue workflows or Step Functions chain crawlers and jobs together, triggered on a schedule or by S3 Event Notifications via EventBridge for near-real-time ingestion.
Example Glue job snippet (PySpark, simplified) converting JSON to partitioned Parquet:
datasource = glueContext.create_dynamic_frame.from_catalog(
database="raw_zone", table_name="orders_json"
)
df = datasource.toDF()
df = df.withColumn("year", year("order_date")) \
.withColumn("month", month("order_date"))
df.write.mode("append") \
.partitionBy("year", "month") \
.parquet("s3://company-datalake/processed/orders/")
Trade-offs to weigh:
- Glue crawlers are convenient but imprecise. They can misinfer types (e.g., treating a numeric ID as
bigintin one partition andstringin another), causing schema conflicts. Many teams define table schemas explicitly via Terraform/CloudFormation (AWS::Glue::Table) instead of relying purely on crawlers, then run crawlers only for partition discovery. - Glue job cost is DPU-hour based (billed per second, minimum 1 minute). A single DPU provides 4 vCPUs and 16 GB RAM; jobs auto-scale with Auto Scaling for Glue but still need tuning of worker type (
G.1X,G.2X,G.025Xfor smaller jobs) to avoid overpaying for idle capacity. - Glue Studio gives a visual, low-code ETL builder, useful for simpler transforms but limiting for complex business logic compared to hand-written Spark.
Querying with Amazon Athena
Amazon Athena lets you run standard SQL directly against data in S3 using the Glue Catalog for schema, with no cluster to provision and billing based purely on bytes scanned. It's built on Trino (formerly Presto SQL) and supports ANSI SQL, window functions, complex joins, and increasingly, Athena for Apache Spark for notebook-style workloads.
Example query against the curated orders table:
SELECT region, SUM(order_total) AS revenue
FROM gold.orders
WHERE year = 2026 AND month = 1
GROUP BY region
ORDER BY revenue DESC;
Because year and month are partition columns, Athena skips every file outside January 2026 before scanning a single byte of actual data — this is why partition design is treated as a first-class architecture decision, not an afterthought.
Key operational characteristics:
- Workgroups let you set per-team query limits (bytes scanned per query, cost controls) and route billing/metrics separately — useful for multi-tenant analytics environments.
- Result caching: Athena automatically reuses results for identical queries run within 24 hours at no additional scan cost, provided the underlying data hasn't changed.
- CTAS (CREATE TABLE AS SELECT) is the standard way to materialize a query result as a new Parquet table, often used to build the Gold zone from Silver zone data entirely in SQL instead of Spark.
- Federated Query: Athena connectors (Lambda-based) can query Amazon RDS, DynamoDB, or on-prem JDBC sources alongside S3 data in a single query, though this trades off the cost-per-scan simplicity for Lambda invocation overhead.
- Concurrency limits: Athena enforces per-account concurrent query limits (default values vary by Region, adjustable via Service Quotas), which matters for high-concurrency BI dashboards.
Choosing a Query Engine: Athena vs. Alternatives
No single engine is right for every workload on a data lake — the choice depends on query patterns, latency requirements, and team skillset.
| Engine | Best for | Pricing model | Operational overhead |
|---|---|---|---|
| Athena | Ad hoc SQL, BI dashboards, infrequent queries | Per TB scanned | None (serverless) |
| Redshift Spectrum | Joining lake data with warehouse tables | Redshift cluster + per-TB scan | Cluster management |
| EMR (Spark/Presto) | Heavy ETL, ML feature pipelines, long jobs | Per instance-hour | Cluster/config tuning |
| Athena for Spark | Interactive notebooks, iterative analysis | Per DPU-hour | Low (serverless) |
A common pattern is to use Glue for scheduled batch ETL, Athena for self-service SQL analytics, and EMR or Athena for Spark when a workload needs iterative processing, custom libraries, or machine learning feature engineering that doesn't map cleanly to SQL.
Redshift Spectrum deserves a specific mention: if your organization already runs Amazon Redshift for a warehouse, Spectrum lets Redshift query S3-resident lake data directly and join it with native Redshift tables — useful when you want one SQL endpoint for both warehouse and lake data, at the cost of needing a running Redshift cluster rather than Athena's pure serverless model.
Security, Governance, and Access Control
Centralizing data in one S3-based lake concentrates risk, so access control has to be deliberate rather than left to default bucket policies. AWS Lake Formation is AWS's purpose-built answer: it sits on top of Glue and S3 and provides column-level, row-level, and tag-based access control, rather than the all-or-nothing access that raw IAM/S3 bucket policies give you.
Practical governance layers:
- IAM policies control who can call Athena, Glue, and S3 APIs at a coarse level.
- Lake Formation permissions grant fine-grained access (e.g., analysts can query
ordersbut not see thecustomer_ssncolumn) enforced at query time regardless of which engine (Athena, EMR, Redshift Spectrum) issues the query. - S3 bucket policies and VPC endpoints restrict network-level access, keeping Athena/S3 traffic off the public internet when queries originate from within a VPC.
- AWS KMS encryption (SSE-KMS) on S3 objects, combined with Glue catalog encryption, covers data-at-rest requirements for regulated workloads (HIPAA, PCI-DSS).
- AWS CloudTrail and Athena query logging provide the audit trail — who queried what, when — often a compliance requirement rather than a nice-to-have.
The trade-off: Lake Formation adds real governance value but also real complexity — permission grants must be modeled carefully, and debugging "why can't this user see this table" issues takes some familiarity with how Lake Formation's grant model interacts with underlying IAM.
Cost Considerations and Common Pitfalls
The S3/Glue/Athena stack is cheap by default compared to a provisioned warehouse, but costs creep in through a few predictable patterns, and it's worth naming them directly.
- Unpruned Athena queries:
SELECT *without partition filters on a large unpartitioned table scans the entire dataset every time — the most common source of surprise Athena bills. - Small files from streaming ingestion: tools like Amazon Kinesis Data Firehose writing directly to S3 every 60 seconds generate thousands of small objects; a periodic Glue compaction job (merging small files into larger Parquet files) is usually necessary.
- Crawler over-scheduling: running Glue crawlers hourly on tables that change daily wastes DPU-hours for no benefit; schedule crawlers to match actual data arrival patterns.
- Catalog sprawl: without naming conventions and a clear database-per-zone structure (
raw,staging,gold), the Glue Catalog becomes hard to govern as table counts grow into the hundreds. - Glue job over-provisioning: defaulting every job to 10
G.2Xworkers regardless of actual data volume is a common way to overspend; right-sizing against actual input data size pays off quickly.
Key Takeaways
- A data lake on AWS separates storage (S3) from compute (Athena, Glue, EMR), letting each scale and bill independently.
- Partitioning and columnar formats (Parquet/ORC) are the primary levers for controlling Athena query cost, since Athena bills per byte scanned.
- AWS Glue provides both the metadata catalog (via crawlers) and serverless ETL (via Spark jobs), but crawler-inferred schemas should be validated, not blindly trusted.
- Amazon Athena is best for ad hoc and BI-style SQL queries; heavier, iterative, or ML-oriented workloads are often better served by EMR or Athena for Spark.
- Table formats like Apache Iceberg, Hudi, or Delta Lake add ACID transactions and row-level updates/deletes on top of S3, which plain Parquet-on-S3 cannot support.
- AWS Lake Formation adds fine-grained, column- and row-level access control across engines — important once a lake has multiple consumer teams with different access needs.
- Most real-world cost problems trace back to small files, unpruned queries, or over-provisioned Glue jobs rather than the architecture itself.
Frequently Asked Questions
Is Athena a replacement for Amazon Redshift?
No, they solve different problems: Athena is a serverless, pay-per-query engine best for ad hoc or infrequent SQL over S3, while Redshift is a provisioned (or serverless-mode) data warehouse optimized for complex, high-concurrency, repeated analytical workloads. Many architectures use both — Athena for lake exploration, Redshift (often via Spectrum) for curated, frequently-queried warehouse tables.
Do I need Apache Iceberg or Delta Lake for a basic data lake?
Not necessarily — plain Parquet files on S3, cataloged in Glue, work fine for append-only or batch-rewritten datasets. You need a table format like Iceberg, Hudi, or Delta Lake once you require row-level updates/deletes, time travel, or concurrent writers, since raw Parquet-on-S3 has no native transaction support.
How does Athena pricing actually work?
Athena charges $5 per TB of data scanned in most US Regions (AWS pricing, as of 2026), rounded up to the nearest MB per query, with a minimum of 10 MB per query. Partition pruning, columnar formats, and compression all reduce the bytes scanned and therefore the cost, which is why storage layout directly drives query spend.
Can Glue crawlers handle schema evolution automatically?
Partially — crawlers can detect new columns or partitions on subsequent runs and update the Glue Catalog, but they can also misinterpret type changes (e.g., an integer column that starts receiving decimal values) and create schema conflicts across partitions. For production pipelines, many teams define schemas explicitly via infrastructure-as-code and use crawlers mainly for partition discovery.
What's the difference between Glue ETL jobs and Glue DataBrew?
Glue ETL jobs are code-based (PySpark/Scala) pipelines suited to complex, repeatable transformations at scale, while AWS Glue DataBrew is a visual, no-code tool aimed at analysts cleaning and profiling data interactively. DataBrew is good for exploratory data prep; Glue jobs are better for production pipelines that need version control and testing.
How do I prevent runaway Athena costs across a large team?
Use Athena workgroups to set per-query or per-workgroup data-scan limits, enforce partitioned/columnar table designs, and route high-frequency dashboard queries through cached or pre-aggregated Gold-zone tables instead of scanning raw data repeatedly. Combine this with AWS Budgets alerts on Athena's cost category for early warning.
Is S3 alone enough for a data lake, or do I always need Glue?
Technically Athena can query S3 data using manually created tables without crawlers, but you still need the Glue Data Catalog (or a compatible Hive Metastore) as the metadata layer — there's no way around having some catalog. What's optional is whether you use Glue crawlers and ETL jobs specifically, versus managing schema manually or running Spark elsewhere (e.g., EMR) while still registering tables in the same Glue Catalog.
Draw this in seconds with draft1. Describe your architecture in plain English and draft1 generates an editable AWS/cloud diagram plus documentation — no dragging boxes around. Try it free.
Draw this in ~20 seconds
Describe your own version of this architecture and draft1 generates an editable draw.io diagram — boxes, arrows, labels, the lot.
Generate this diagram free ➔Free demo — no signup. Then 3 free diagrams with an account, no card.