Skip to content

Software Quality

Monitoring and observability

Monitoring and observability read what a running system is actually doing from the signals it emits in production. Monitoring is the practice of instrumenting the system so those signals reach humans, alerts, and automation. Observability adds the discipline that telemetry should answer questions not predicted at deployment time, typically through high-cardinality structured logs, distributed traces, and metrics with rich labels. The Google SRE book documents the practice as Google runs it (Beyer et al. 2016)1; Observability Engineering (Majors et al. 2022)2 covers the high-cardinality discipline.

The method takes one shape, observing live signals, but runs in two settings:

  • Infrastructure the operator runs. Services on machines the operator owns or rents: a uniform fleet the operator can scale, redeploy, and attach a debugger to.
  • Devices the operator does not control. A browser app, a mobile app, a desktop program, or an IoT device runs somewhere the operator cannot reach.

What it catches

  • Latency regressions. A new release inflates p99 by 30%; the dashboard reads it and the alert fires.
  • Error-rate spikes and saturation. A rising error ratio or a filling queue shows on the measured signal itself, with no inference step.
  • Crashes and unhandled exceptions. Error and crash telemetry captures the stack trace, request, and frequency of each exception thrown in production: a definite failure event, the errors oracle. Dedicated trackers (Sentry, Rollbar, Crashlytics) group these exceptions and alert on them.
  • SLO violations. When a service-level objective is stated in measurable telemetry, monitoring reports whether the error budget is intact.
  • Runtime invariant violations. Structured events that carry invariants (cache hit ratio, queue depth, replication lag) let the dashboard catch drift a test suite cannot. When the invariant is a formal temporal property (an LTL or MTL spec such as "every acquire is eventually released"), a monitor compiled over the live event trace checks it continuously. This is runtime verification, the formal end of the same live invariant check.
  • Anomalies and unknown unknowns. A sudden shift in user-agent mix, traffic shape, or request size is hard to test for and easy to see on a dashboard; ad-hoc queries against rich telemetry answer questions no dashboard anticipated.

What monitoring does not catch is anything before the problem reaches production.

Infrastructure the operator runs: the SRE practice

For a service fleet the environment is uniform and reproducible, so the open question is what to instrument first.

Framework Metrics Layer it watches
USE (Gregg 2013)3 utilization, saturation, errors each resource: CPU, disk, network link, lock, queue
Four Golden Signals (Ewaschuk 2016)4 latency, traffic, errors, saturation the user-facing service (the SLO-level summary)
RED (Wilkie 2018)5 rate, errors, duration each request-driven service or endpoint (Golden Signals minus saturation)

RED is the request-path projection of the Golden Signals and USE the resource-path projection: RED on every service plus USE on every resource covers both, and the Golden Signals roll up to the user-facing SLO. Above the signals sit the objectives themselves: an SLO is a bound on a distribution tracked as an error budget, and health invariants are watched as live predicates.

Devices the operator does not control: field telemetry

The method is still observation of live signals, but the loss of control over the environment changes what that observation can do:

  • The telemetry is the only window. There is no second look: the operator cannot reproduce a user's exact device, OS build, locale, network path, or hardware, and cannot attach a debugger after the fact. For a service, monitoring is the last line behind everything reproducible upstream; for field software it is sometimes the only line.
  • The install base is the real input distribution. A service fleet is uniform by design; an install base is not. Field telemetry reports the device, OS-version, locale, screen-size, and connection-type mix that actually exists, the input distribution no test matrix enumerates up front. That distribution feeds back into what to test, making field telemetry a source of test inputs as much as a failure detector.
  • Collection is sampled, batched, and consent-bound. Events are sampled to control volume, queued and sent later over networks that drop them, and gated by user consent. Under a consent regime such as GDPR, anonymous aggregates are often all an operator may collect without opt-in, so the oracle checks a distribution over a population rather than a single session's trace, and a late or missing event is the normal case, not an incident.
  • Crashes are harder to read. A field crash arrives without the context a server stack trace carries: it must be symbolicated against the right build, the install base spans many versions at once, and the payload is often a minidump rather than a live process.

The signals differ per platform: browsers report Core Web Vitals and JavaScript errors through RUM; mobile reports crash-free-session rate, ANRs, and the store vitals (Play Console, App Store Connect); desktop reports crashes through Breakpad/Crashpad or Windows Error Reporting; IoT is the hard case, where intermittent connectivity forces store-and-forward and constrains how much can be sent at all. Product analytics (funnels, feature adoption, conversion) reuses the same pipelines but answers a different question.

Tools

Metrics

  • Prometheus + Grafana — open-source de facto pair for pull-based metrics and dashboards.
  • Datadog, New Relic, Splunk Observability — managed metrics + APM; commercial.
  • VictoriaMetrics, Thanos, Mimir — long-term storage layers for Prometheus.
  • OpenTelemetry Metrics — vendor-neutral metrics API.

Logs

Tracing and APM

  • OpenTelemetry — the vendor-neutral standard; tracing, metrics, logs.
  • Jaeger, Zipkin, Tempo — open-source trace backends.
  • Datadog APM, New Relic, Dynatrace, AppDynamics — commercial.

Field telemetry (RUM, mobile, desktop, IoT)

  • Browser RUM: Sentry, Datadog RUM (commercial), New Relic Browser (commercial), Cloudflare Web Analytics — real-user metrics from the browser.
  • Mobile: Firebase Crashlytics, Sentry, Embrace (commercial) — crash and vitals reporting from a shipped app.
  • Desktop / native crash reporting: Google Breakpad / Crashpad, Windows Error Reporting, Mozilla crash-stats — symbolicated crash and minidump collection.

Alerting and on-call

  • PagerDuty, Opsgenie, Incident.io (all commercial), Grafana OnCall — paging and on-call rotation.
  • Alertmanager (Prometheus's alert router).

Continuous profiling

An always-on profiler emits a production signal like any other; see profiling for the technique.

  • Pyroscope, Parca, Datadog Continuous Profiler and Google Cloud Profiler (both commercial).

When to use, when not

Use:

  • Every production system. The minimum bar is the Golden Signals per service, an alert on SLO violation, and a place to find logs.
  • Anywhere business-critical invariants need a runtime watcher (queue depth, replication lag, cache hit ratio, error-budget burn rate).
  • High-cardinality systems where the failure modes are unknown at design time. The observability discipline of wide events and rich labels lets operators answer unforeseen questions.

Don't:

  • For pre-merge verification. Monitoring is downstream of every other method.
  • For low-volume internal tools where the operator is the user. The cost of running the stack outweighs the value.
  • Without an alerting strategy. Dashboards without alerts detect nothing until somebody looks at them; alerts with nobody on call go unanswered.
  • Without verifying the alerts themselves. An alarm threshold and an SLO definition are code: an alarm nobody has fired on purpose is a claim about detection, not evidence of it. A synthetic breach shows it fires, and the alert history grouped by signature shows which definitions produce the pages.

Evidence

Monitoring resists measurement against a control because the telemetry is also the instrument: a service with no telemetry emits no record of the failures it had, so there is nothing to compare a monitored service against. The case for monitoring is definitional rather than statistical — a failure that produces no observable signal cannot be diagnosed or fixed.

Classification

  • Quality dimensions: Reliability, Performance, Functionality (when invariants fire).
  • Area: Production services, microservices, SaaS, on-call operations, and software running on devices the operator does not control (browsers, mobile, desktop, IoT) reporting field telemetry; anywhere live signals are observed against SLOs or field aggregates.
  • Guarantee: Heuristic.

Referenced by

References

  1. Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds. 2016. Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. https://sre.google/sre-book/. ↩

  2. Majors, Charity, Liz Fong-Jones, and George Miranda. 2022. Observability Engineering: Achieving Production Excellence. O'Reilly. ↩

  3. Gregg, Brendan. 2013. "Thinking Methodically about Performance." Communications of the ACM 56 (2): 45–51. https://doi.org/10.1145/2408776.2408791. ↩

  4. Ewaschuk, Rob. 2016. "Monitoring Distributed Systems." In Site Reliability Engineering: How Google Runs Production Systems, edited by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. O'Reilly. https://sre.google/sre-book/monitoring-distributed-systems/. ↩

  5. Wilkie, Tom. 2018. The RED Method: Patterns for Instrumentation and Monitoring. Talk, GrafanaCon EU. https://grafana.com/files/grafanacon_eu_2018/Tom_Wilkie_GrafanaCon_EU_2018.pdf. ↩