# Monitoring and observability

Monitoring and observability read what a running system is actually doing from
the signals it emits in production. *Monitoring* is the practice of instrumenting
the system so those signals reach humans, alerts, and automation.
*Observability* adds the discipline that telemetry should answer *questions not
predicted at deployment time*, typically through high-cardinality structured
logs, distributed traces, and metrics with rich labels. The Google SRE book is
the canonical industrial reference (Beyer et al. 2016)[^beyer2016]; *Observability Engineering*
(Majors et al. 2022)[^majors2022] covers the modern high-cardinality discipline.

The method takes one shape, observing live signals, but runs in two settings,
and which one you are in decides what observation can do:

- **Infrastructure you operate.** Services on machines you own or rent: a
  uniform fleet you can scale, redeploy, and attach a debugger to. The
  classical SRE practice applies (SLOs, the named signal frameworks, health
  invariants).
- **Devices you don't control.** A browser app, a mobile app, a desktop
  program, or an IoT device runs somewhere the operator cannot reach. Field
  telemetry is often the *only* record of a failure, the install base is the
  real input distribution, and collection is sampled and consent-bound.

## What it catches

- **Latency regressions.** A new release inflates p99 by 30%; the dashboard
  reads it, the alert fires.
- **Error-rate spikes and saturation.** A rising error ratio or a filling queue
  shows on the measured signal itself, with no inference step (Beyer et al. 2016)[^beyer2016].
- **Crashes and unhandled exceptions.** Error and crash telemetry captures the
  stack trace, request, and frequency of each exception thrown in production: a
  definite failure event, the errors oracle. Dedicated trackers (Sentry,
  Rollbar, Crashlytics) specialize in the grouping and alerting.
- **SLO violations.** When a service-level objective is stated in measurable
  telemetry, monitoring reports whether the error budget is intact.
- **Runtime invariant violations.** Structured events that carry invariants
  (cache hit ratio, queue depth, replication lag) let the dashboard catch drift
  a test suite cannot. When the invariant is a *formal temporal property* (an
  LTL or MTL spec such as "every acquire is eventually released"), a monitor
  compiled over the live event trace checks it continuously. This is **runtime
  verification**, the formal end of the same live invariant check.
- **Anomalies and unknown unknowns.** A sudden shift in user-agent mix, traffic
  shape, or request size is hard to test for and easy to see on a dashboard;
  ad-hoc queries against rich telemetry answer questions no dashboard
  anticipated, the promise of observability over plain monitoring.

What monitoring does **not** catch is anything before the problem reaches
production.

## Infrastructure you operate: the SRE practice

For a service fleet the environment is uniform and reproducible, so the open
question is *what to instrument first*.

| Framework | Metrics | Layer it watches |
| --- | --- | --- |
| [USE](https://www.brendangregg.com/usemethod.html) (Gregg 2013)[^gregg2013] | utilization, saturation, errors | each resource: CPU, disk, network link, lock, queue |
| [Four Golden Signals](https://sre.google/sre-book/monitoring-distributed-systems/) (Beyer et al. 2016)[^beyer2016] | latency, traffic, errors, saturation | the user-facing service (the SLO-level summary) |
| [RED](https://grafana.com/blog/the-red-method-how-to-instrument-your-services/) (Wilkie 2018)[^wilkie2018] | rate, errors, duration | each request-driven service or endpoint (Golden Signals minus saturation) |

RED is the request-path projection of the Golden Signals and USE the
resource-path projection: RED on every service plus USE on every resource covers
both, and the Golden Signals roll up to the user-facing SLO. Above the signals
sit the objectives themselves, an SLO stated as a bound on a distribution and
tracked as an error budget, and the health invariants (queue depth, replication
lag, cache hit ratio) watched as live predicates.

## Devices you don't control: field telemetry

A service runs on infrastructure the operator owns; field software runs where
the operator cannot reach. The method is still observation of live signals, but
the loss of control over the environment changes what that observation can do:

- **The telemetry is the only window.** There is no second look: the operator
  cannot reproduce a user's exact device, OS build, locale, network path, or
  hardware, and cannot attach a debugger after the fact. Whatever the event
  captured is the entire record of a failure. For a service, monitoring is the
  last line behind everything reproducible upstream; for field software it is
  sometimes the *only* line, because the failing condition cannot be recreated
  in a lab.
- **The install base is the real input distribution.** A service fleet is
  uniform by design; an install base is not. Field telemetry reports the device,
  OS-version, locale, screen-size, and connection-type mix that actually exists,
  the input distribution no test matrix enumerates up front. That distribution
  feeds back into what to test, making field telemetry a source of test inputs
  as much as a failure detector.
- **Collection is sampled, batched, and consent-bound.** Events are sampled to
  control volume, queued and sent later over networks that drop them, and gated
  by user consent. Anonymous aggregate statistics are frequently the only lawful
  signal (opt-in telemetry, GDPR), so the oracle checks a distribution over a
  population rather than a single session's trace, and a late or missing event is
  the normal case, not an incident.
- **Crashes are harder to read.** A field crash arrives without the context a
  server stack trace carries: it must be symbolicated against the right build,
  the install base spans many versions at once, and the payload is often a
  minidump rather than a live process.

The named signals differ per platform: browsers report Core Web Vitals and
JavaScript errors through RUM; mobile reports crash-free-session rate, ANRs, and
the store vitals (Play Console, App Store Connect); desktop reports crashes
through Breakpad/Crashpad or Windows Error Reporting; IoT is the hard case,
where intermittent connectivity forces store-and-forward and constrains how much
can be sent at all. Product analytics (funnels, feature adoption, conversion)
reuses the same pipelines but answers a different question and is out of scope
here.

## Tools

### Metrics

- **[Prometheus](https://prometheus.io/)** + **[Grafana](https://grafana.com/)** — open-source de facto pair for
  pull-based metrics and dashboards.
- **Datadog**, **New Relic**, **Splunk Observability** — managed
  metrics + APM; commercial.
- **[VictoriaMetrics](https://victoriametrics.com/)**, **[Thanos](https://thanos.io/)**, **[Mimir](https://grafana.com/oss/mimir/)** — long-term storage
  layers for Prometheus.
- **[OpenTelemetry](https://opentelemetry.io/) Metrics** — vendor-neutral metrics API.

### Logs

- **[Elastic Stack](https://www.elastic.co/elastic-stack)** (Elasticsearch + Kibana), **Splunk**,
  **[Loki](https://grafana.com/oss/loki/) + Grafana**, **[OpenObserve](https://openobserve.ai/)**, **[ClickHouse](https://clickhouse.com/)** — log search
  and aggregation.
- **[structlog](https://www.structlog.org/)** (Python), **[slog](https://pkg.go.dev/log/slog)** (Go 1.21+), **[tracing-subscriber](https://github.com/tokio-rs/tracing)**
  (Rust), **[pino](https://getpino.io/)** / **[winston](https://github.com/winstonjs/winston)** (Node) — structured logging
  libraries.

### Tracing and APM

- **[OpenTelemetry](https://opentelemetry.io/)** — the vendor-neutral standard; tracing,
  metrics, logs.
- **[Jaeger](https://www.jaegertracing.io/)**, **[Zipkin](https://zipkin.io/)**, **[Tempo](https://grafana.com/oss/tempo/)** — open-source trace backends.
- **Datadog APM**, **New Relic**, **Dynatrace**, **AppDynamics** —
  commercial.

### Field telemetry (RUM, mobile, desktop, IoT)

- **Browser RUM:** **Sentry**, **Datadog RUM**, **New Relic Browser**,
  **Cloudflare Web Analytics**. Real-user metrics for the page load
  and beyond.
- **Mobile:** **Firebase Crashlytics**, **Sentry**, **Embrace**;
  crash-free rate, ANRs, and store vitals (Play Console, App Store
  Connect).
- **Desktop / native crash reporting:** **Google Breakpad** /
  **Crashpad**, **Windows Error Reporting**, **Mozilla crash-stats**.
  Symbolicated crash and minidump collection from an uncontrolled
  install base.

### Alerting and on-call

- **PagerDuty**, **Opsgenie**, **Grafana OnCall**, **Incident.io**.
- **[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)** (Prometheus's alert router).

### Continuous profiling

Always-on profilers in production; see [profiling](https://quality.stereobooster.com/profiling.md)
for the technique.

- **[Pyroscope](https://grafana.com/oss/pyroscope/)**, **[Parca](https://www.parca.dev/)**, **Datadog Continuous Profiler**,
  **Google Cloud Profiler**.

## When to use, when not

**Use:**

- Every production system. The minimum bar is *Golden Signals*
  (latency, traffic, errors, saturation) per service, alert on
  SLO violation, and a place to find logs.
- Anywhere business-critical invariants need a runtime watcher
  (queue depth, replication lag, cache hit ratio, error-budget
  burn rate).
- High-cardinality systems where the failure modes are unknown at
  design time. The observability discipline, wide events and rich
  labels, is what lets operators answer unforeseen questions.

**Don't:**

- For pre-merge verification. Monitoring is downstream of every
  other method.
- For low-volume internal tools where the operator *is* the user.
  The cost of running the stack outweighs the value.
- Without an alerting strategy. Dashboards without alerts are
  monitoring theater; alerts without an on-call rotation are
  alert spam.
- Without verifying the alerts themselves. An alarm threshold and an
  SLO definition are code, and usually the least-tested code in a
  system: an alarm nobody has fired on purpose is a claim about
  detection, not evidence of it. A synthetic breach shows it fires,
  and the alert history grouped by *signature* shows which
  definitions produce the pages.

## Evidence

The empirical literature on monitoring is *industrial reports*: no controlled
study compares monitored and unmonitored services, since monitoring is a
baseline assumption across the literature rather than a treatment. The case for
monitoring is definitional — a failure that produces no observable signal cannot
be diagnosed or fixed.

## Classification

- **Quality dimensions:** Reliability, Performance, Functionality (when invariants fire).
- **Area:** Production services, microservices, SaaS, on-call operations, and software running on devices the operator does not control (browsers, mobile, desktop, IoT) reporting field telemetry; anywhere live signals are observed against SLOs or field aggregates.
- **Guarantee:** Heuristic.

## Referenced by

- [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions
- [Effect scope](https://quality.stereobooster.com/effect.md) · The axes
- [Chaos engineering](https://quality.stereobooster.com/chaos-engineering.md) · Methods
- [Contracts and runtime assertions](https://quality.stereobooster.com/contracts-and-runtime-assertions.md) · Methods
- [Dead-code detection](https://quality.stereobooster.com/dead-code-detection.md) · Methods
- [Load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md) · Methods
- [Parallel run](https://quality.stereobooster.com/parallel-run.md) · Methods
- [Safety analysis](https://quality.stereobooster.com/safety-analysis.md) · Methods
- [Temporal-logic falsification](https://quality.stereobooster.com/temporal-logic-falsification.md) · Methods
- [Termination analysis](https://quality.stereobooster.com/termination-analysis.md) · Methods
- [Testing autonomous and cyber-physical systems](https://quality.stereobooster.com/testing-autonomous-systems.md) · Methods
- [Testing machine-learning systems](https://quality.stereobooster.com/testing-ml-systems.md) · Methods
- [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI

## References

[^beyer2016]: Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds. 2016. *[Site Reliability Engineering: How Google Runs Production Systems](https://sre.google/sre-book/)*. O'Reilly. <https://sre.google/sre-book/>.
[^majors2022]: Majors, Charity, Liz Fong-Jones, and George Miranda. 2022. *Observability Engineering: Achieving Production Excellence*. O'Reilly.
[^gregg2013]: Gregg, Brendan. 2013. "[Thinking Methodically about Performance](https://cacm.acm.org/magazines/2013/2/160167-thinking-methodically-about-performance/)." *Communications of the ACM* 56 (2): 45–51. <https://doi.org/10.1145/2408776.2408791>.
[^wilkie2018]: Wilkie, Tom. 2018. *[The RED Method: Patterns for Instrumentation and Monitoring](https://grafana.com/files/grafanacon_eu_2018/Tom_Wilkie_GrafanaCon_EU_2018.pdf)*. Talk, GrafanaCon EU. [https://grafana.com/files/grafanacon\\\_eu\\\_2018/Tom\\\_Wilkie\\\_GrafanaCon\\\_EU\\\_2018.pdf](https://grafana.com/files/grafanacon\_eu\_2018/Tom\_Wilkie\_GrafanaCon\_EU\_2018.pdf).

## Acronyms

- ANR — application not responding
- APM — application performance monitoring
- LTL — linear temporal logic
- MTL — metric temporal logic
- RED — Rate, Errors, Duration (the request-driven metrics method)
- RUM — real-user monitoring
- SLO — service level objective
- SRE — site reliability engineering
- USE — Utilization, Saturation, Errors (Brendan Gregg's method)