Observability Stack Architecture

What this diagram shows

This diagram represents the flow of observability signals (metrics, logs, traces) from instrumented applications through collection and storage layers into a unified dashboard and alerting system. Applications emit telemetry using OpenTelemetry SDKs or agents, which route metrics to Prometheus, logs to Loki, and traces to Tempo. These three backends store their respective signals independently. Prometheus evaluates alert rules and fires notifications, while Grafana queries all three data sources, dashboards surface service-level objectives (SLOs) calculated from the stored data, and operators respond to firing alerts. The architecture separates concerns by data type while maintaining single-pane-of-glass visibility.

Key components

• Application Instrumentation — Emits metrics, logs, and spans using OpenTelemetry SDKs, libraries, or auto-instrumentation agents embedded in services.
• OpenTelemetry Collector — Receives signals from instrumented applications via OTLP protocol, applies transformations and filtering, then exports to backend storage systems.
• Prometheus — Scrapes or receives metrics via push, stores time-series data, and evaluates alert rules against metric thresholds to generate alerts.
• Loki — Stores application and infrastructure logs indexed by labels, allowing log queries and filtering without full-text index overhead.
• Tempo — Stores distributed traces with span relationships and timing information, enabling service dependency mapping and latency analysis.
• Grafana — Queries Prometheus, Loki, and Tempo simultaneously to surface metrics, logs, traces, and SLO compliance in interactive dashboards.
• Alerting and Notification — Receives alerts fired by Prometheus or Grafana alert rules and routes them to incident management systems, chat platforms, or PagerDuty.

When to use it

Use this pattern when you need to correlate metrics, logs, and traces across distributed systems without vendor lock-in. It works well for teams already running Kubernetes or container infrastructure, adopting OpenTelemetry standards, and wanting to avoid separate point solutions for each signal type. Choose it when SLO monitoring and alert-driven incident response are priorities, and your team has capacity to operate multiple stateful backends.

Common mistakes

• Treating OpenTelemetry Collector as optional and having each application push directly to backends, which bypasses crucial signal processing, sampling, and reliability.
• Storing all logs indefinitely in Loki without retention policies or partitioning strategies, causing unbounded disk growth and expensive query latency.
• Configuring alert rules only on metrics without correlating them to trace data or logs, leading to alerts firing without enough context to troubleshoot root cause.

Adapting it to your system

Replace the generic application box with your actual services: microservices, Lambda functions, or containerized workloads. Adjust OpenTelemetry Collector configuration for your environment (Kubernetes DaemonSet, sidecar, or gateway mode). Specify Prometheus scrape targets matching your infrastructure. Define Loki label schema around your deployment boundaries (namespace, service, environment). Add storage backends appropriate to your scale: S3 for Loki, object storage for Tempo, or local PVs for Prometheus in non-production. Tailor SLO queries to your actual error budgets and business objectives.