Building a Real-Time Analytics Pipeline on AWS

A real-time analytics pipeline on AWS typically combines Amazon Kinesis Data Streams for ingestion, a stream processor like Managed Service for Apache Flink for transformation, and a serving layer such as OpenSearch or DynamoDB for sub-second queries. The result is a system that can turn clickstreams, IoT telemetry, or transaction events into dashboards and alerts within seconds instead of hours.

This article walks through a concrete, buildable [architecture:](https://www.draft1.ai/blog/message-queue-architecture-kafka-vs-sqs-vs-rabbitmq) ingestion, processing, storage, and serving, with the trade-offs AWS engineers actually run into — shard math, Flink checkpointing, cost at scale, and when a Kinesis pipeline is overkill compared to simpler options like Lambda-only or Kafka on MSK. By the end you'll have a reference design you can adapt, not just a conceptual diagram.

What "real-time" actually means in an AWS context

Real-time analytics on AWS almost always means "near real-time" — latencies measured in seconds, not true microsecond streaming. Most business use cases (fraud scoring, operational dashboards, anomaly detection) tolerate 1–10 second end-to-end latency, which is what Kinesis-based architectures deliver reliably.

It's worth separating three tiers before you design anything:

• Batch (minutes to hours): S3 + Glue + Athena/Redshift. Fine for daily reporting.
• Near real-time (seconds): Kinesis Data Streams or MSK + stream processing + a fast serving store. This is the focus of this article.
• Hard real-time (sub-millisecond): specialized systems (FPGA trading engines, embedded control loops) — AWS general-purpose services aren't built for this, full stop.

Most teams asking for "real-time analytics" actually need near real-time. Confirming this with stakeholders early avoids over-engineering a system for latency requirements nobody actually has.

Core architecture: ingestion with Kinesis Data Streams

Amazon Kinesis Data Streams is a managed, partitioned log service that ingests high-throughput event data and retains it for ordered, replayable consumption. It's the backbone of most AWS real-time pipelines because it decouples producers from consumers and supports multiple parallel readers.

Key mechanics worth knowing before you provision anything:

• A stream is divided into shards. Each shard supports 1 MB/sec or 1,000 records/sec ingest, and 2 MB/sec output (shared across consumers unless using enhanced fan-out).
• Partition keys determine shard assignment via MD5 hashing. Skewed partition keys (e.g., using customer_id when 80% of traffic comes from one customer) cause "hot shards" — a common and painful production bug.
• Enhanced fan-out gives each registered consumer a dedicated 2 MB/sec pipe per shard instead of sharing bandwidth, at extra cost (~$0.015/consumer-shard-hour plus data transfer). Use this when you have more than one or two downstream consumers reading the same stream.
• Retention is 24 hours by default, extendable to 365 days (extended retention incurs additional cost), which lets you replay data for reprocessing or recovery.
• On-demand mode auto-scales shards based on throughput, trading predictable cost for operational simplicity. Provisioned mode is cheaper at steady, well-understood volumes but requires manual or auto-scaling-policy shard management.

For most pipelines, start with on-demand mode until you understand your traffic pattern, then switch to provisioned once volume stabilizes — on-demand pricing runs roughly 2x provisioned-equivalent cost at scale (AWS pricing pages, as of 2026).

Alternative ingestion: Amazon MSK

If your organization already runs Kafka, or you need features like compacted topics, exactly-once semantics across heterogeneous consumer groups, or very long retention without extra cost scaling, Amazon MSK (Managed Streaming for Kafka) is the alternative. It requires more operational awareness (broker sizing, partition rebalancing) but gives you the full Kafka ecosystem (Kafka Connect, ksqlDB-compatible tooling via third parties).

Processing layer: Flink, Lambda, or KDA

Stream processing is where raw events become aggregates, enrichments, or alerts — and the right tool depends on windowing complexity and state size. AWS offers three realistic choices, each with sharp trade-offs.

AWS Lambda consuming directly from a Kinesis stream works well for stateless transformations and routing — parsing JSON, filtering, enriching with a DynamoDB lookup, then writing out. It's cheap and simple but struggles with windowed aggregations (e.g., "rolling 5-minute average per user") because Lambda has no native stateful windowing; you'd have to roll your own state in DynamoDB or ElastiCache, which adds latency and cost.

Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics for Apache Flink) is the right tool when you need true windowed aggregation, event-time processing with watermarks, joins across streams, or complex event pattern detection (CEP). It manages checkpointing to S3, scaling via Kinesis Processing Units (KPUs), and integrates natively with Kinesis and MSK as sources. The cost is operational complexity: you're now managing a Flink application's state backend, checkpoint intervals, and parallelism tuning.

Amazon EMR with Spark Structured Streaming is a reasonable middle ground if your team already has Spark expertise and wants micro-batch semantics (typically 1–10 second batch intervals) rather than Flink's true event-at-a-time model. It's heavier to operate than managed Flink but offers more flexibility for teams with existing Spark pipelines.

A representative data flow for a clickstream analytics use case:

Producers (web/app SDKs)
      │  PutRecords (HTTPS/443)
      ▼
Kinesis Data Streams (ingestion, ordered by partition key)
      │
      ▼
Managed Service for Apache Flink
  - parse + validate events
  - 1-minute tumbling window: event counts per page
  - sessionization: 30-min gap sessions
      │
      ├──► Kinesis Data Firehose ──► S3 (Parquet, partitioned by date/hour) ──► Athena/Redshift Spectrum
      │
      └──► DynamoDB (latest aggregates) ──► API Gateway + Lambda ──► Dashboard (sub-second reads)

This fan-out pattern — one hot path for dashboards, one cold path for historical analysis — is the most common real-world design, because it serves both "what's happening right now" and "show me trends over the last 90 days" from the same source stream without duplicating ingestion logic.

Storage and serving layer

The serving layer determines how fast and how flexibly users can query your processed data, and no single store does both cheap-historical and fast-current well. Pick based on query pattern, not habit.

• DynamoDB: best for point lookups and "latest value" dashboards (e.g., current inventory count, latest sensor reading). Single-digit millisecond reads, but poor for ad hoc analytical queries or large scans.
• Amazon OpenSearch Service: strong for log analytics, full-text search, and near-real-time dashboards (Kibana/OpenSearch Dashboards) with flexible filtering. Costs more to run continuously and requires index lifecycle management to control storage growth.
• Amazon Redshift (with streaming ingestion): Redshift now supports direct streaming ingestion from Kinesis and MSK without a Firehose/S3 hop, materializing data into tables within seconds. Good when downstream users need SQL joins against large historical tables.
• S3 + Athena/Redshift Spectrum: the cheapest long-term store, fed via Kinesis Data Firehose, typically converting to Parquet and partitioning by date/hour for query pruning. Latency here is minutes, not seconds — this is your cold path, not your hot path.

Comparing the processing options

For engineers deciding where to put their transformation logic, here's how the three mainstream choices stack up:

Handling schema, ordering, and exactly-once concerns

Stream processing introduces failure modes batch pipelines don't have: out-of-order events, duplicate delivery, and schema drift — all of which need explicit handling, not hope.

• Ordering: Kinesis guarantees ordering within a shard based on sequence number, not across the whole stream. If strict per-key ordering matters (e.g., all events for order_id=123 must process in order), your partition key must be the entity you need ordered — but this reintroduces the hot-shard risk if that entity's traffic is skewed.
• Exactly-once vs. at-least-once: Kinesis consumers (Lambda, Flink, KCL-based apps) default to at-least-once delivery — a shard split, consumer restart, or Lambda retry can redeliver a record. Flink with checkpointing and idempotent sinks (e.g., upserts keyed by event ID) gets you effectively-once outcomes; Lambda needs you to build idempotency yourself, typically via a DynamoDB conditional write keyed on event ID.
• Late and out-of-order data: event-time windowing (Flink's watermarks) handles clock skew between producers and the pipeline, but you must decide an allowed lateness threshold — data arriving after that window closes either gets dropped or routed to a "late data" side output.
• Schema evolution: use a schema registry (AWS Glue Schema Registry integrates with Kinesis and MSK) so producers and consumers agree on field additions/removals without silent parsing failures. Without this, a single unannounced field rename can silently drop records downstream.

Cost and scaling realities

Real-time pipelines are usually more expensive per-event than batch, and the cost driver is almost always shard count and continuously-running compute, not storage. A rough comparison for a moderate workload (10,000 events/sec average, ~1 KB/event):

The single highest-leverage cost decision is usually whether your Flink/EMR cluster runs continuously or can scale toward zero during quiet periods — unlike Lambda, Flink applications are billed for provisioned capacity whether or not events are flowing, so bursty, low-volume workloads often favor a Lambda-only design even if it means hand-rolling simple windowing logic.

When not to build this

A full Kinesis-plus-Flink pipeline is unnecessary if your actual requirement is a dashboard refreshed every 15 minutes, or your event volume is under a few hundred records per second with no complex windowing. In those cases, simpler options cost less and have far fewer moving parts to operate:

• S3 + EventBridge Scheduler + Athena for periodic batch refreshes.
• DynamoDB Streams + Lambda if your "stream" is really just change-data-capture off a single table.
• Kinesis Data Firehose directly to S3/Redshift (no Data Streams or Flink in between) if you only need near-real-time delivery, not real-time computation — Firehose buffers and batches automatically (configurable buffer: 1–900 seconds or 1–128 MB).

Teams frequently over-build these pipelines because "real-time" sounds impressive in a design review, then spend months tuning Flink checkpoint intervals for a dashboard that's refreshed by a human looking at it twice a day.

Key Takeaways

• Kinesis Data Streams is the standard AWS ingestion backbone for real-time pipelines, with throughput governed by shard count (1 MB/sec in, 2 MB/sec out per shard).
• Choose Lambda for stateless, simple transformations; choose Managed Service for Apache Flink when you need event-time windowing, joins, or complex event processing.
• Partition key selection directly determines whether you get even load distribution or hot-shard bottlenecks — design it around your highest-cardinality, most evenly distributed field.
• Kinesis-based pipelines deliver at-least-once delivery by default; exactly-once outcomes require idempotent sinks and careful checkpointing, not assumptions.
• A hot path / cold path split (DynamoDB or OpenSearch for live dashboards, S3 + Athena for historical analysis) is the most common and most maintainable serving pattern.
• The biggest cost lever is continuously-running compute (Flink KPUs, EMR clusters) — confirm your latency requirement is real before committing to always-on infrastructure.
• For low-volume or infrequent-refresh use cases, Firehose-to-S3 or scheduled batch jobs are simpler and cheaper than a full streaming stack.

Frequently Asked Questions

What's the difference between Kinesis Data Streams and Kinesis Data Firehose?

Kinesis Data Streams is a low-latency, replayable, shard-based log that you consume with custom applications (Lambda, Flink, KCL); Kinesis Data Firehose is a fully managed delivery service that buffers and batches data automatically into destinations like S3, Redshift, or OpenSearch. Use Data Streams when you need to run custom processing logic on events as they arrive; use Firehose when you just need reliable delivery with minimal code.

How many shards do I need for my Kinesis stream?

Divide your peak expected throughput by the per-shard limits (1 MB/sec or 1,000 records/sec, whichever binds first) and round up, then add headroom for traffic spikes and retries. In on-demand mode AWS manages this for you automatically, which is why most teams start there before switching to provisioned mode once they understand steady-state volume.

Is Amazon MSK better than Kinesis for real-time analytics?

Neither is universally better — MSK suits teams already invested in the Kafka ecosystem or needing features like compacted topics and long retention without Kinesis's extended-retention charges, while Kinesis offers lower operational overhead since AWS manages partition/shard scaling more transparently. Teams with no existing Kafka expertise generally find Kinesis faster to get into production.

Can I achieve true exactly-once processing with Kinesis?

Not natively at the stream level — Kinesis delivers at-least-once, so exactly-once outcomes require idempotent processing downstream. Managed Flink combined with idempotent sinks (e.g., upserts keyed on a unique event ID) is the standard way to get effectively-once semantics in practice.

How much latency should I expect end-to-end?

For a well-tuned Kinesis-plus-Flink pipeline feeding DynamoDB or OpenSearch, end-to-end latency (event produced to queryable in the dashboard) typically runs 1–5 seconds under normal load. Latency grows if you add Firehose buffering (configurable 1–900 seconds), large windowing intervals, or if a shard becomes a throughput bottleneck.

Do I need Apache Flink, or is Lambda enough?

Lambda is enough for stateless filtering, enrichment, and routing; you need Flink (or similar) once you require time-based windowed aggregations, joins across multiple streams, or complex event pattern detection. Trying to replicate true windowing logic in Lambda usually means building a fragile, hand-rolled state store, which is exactly the problem Flink's managed state and checkpointing are designed to solve.

What happens if my stream consumer falls behind?

Kinesis retains data for 24 hours by default (extendable up to 365 days), so a lagging consumer can catch up by reading from its last checkpointed sequence number rather than losing data, unlike many fire-and-forget messaging systems. Sustained lag usually signals you need more shards, enhanced fan-out for that consumer, or more parallelism in your processing application.

Draw this in seconds with draft1. Describe your architecture in plain English and draft1 generates an editable AWS/cloud diagram plus documentation — no dragging boxes around. Try it free.