Skip to content

Software Quality

Monitoring and observability

Monitoring and observability read what a running system is actually doing from the signals it emits in production. Monitoring is the practice of instrumenting the system so those signals reach humans, alerts, and automation. Observability adds the discipline that telemetry should answer questions not predicted at deployment time, typically through high-cardinality structured logs, distributed traces, and metrics with rich labels. The Google SRE book is the canonical industrial reference (Beyer et al. 2016)1; Observability Engineering (Majors et al. 2022)2 covers the modern high-cardinality discipline.

The method takes one shape, observing live signals, but runs in two settings, and which one you are in decides what observation can do:

  • Infrastructure you operate. Services on machines you own or rent: a uniform fleet you can scale, redeploy, and attach a debugger to. The classical SRE practice applies (SLOs, the named signal frameworks, health invariants).
  • Devices you don't control. A browser app, a mobile app, a desktop program, or an IoT device runs somewhere the operator cannot reach. Field telemetry is often the only record of a failure, the install base is the real input distribution, and collection is sampled and consent-bound.

What it catches

  • Latency regressions. A new release inflates p99 by 30%; the dashboard reads it, the alert fires.
  • Error-rate spikes and saturation. A rising error ratio or a filling queue shows on the measured signal itself, with no inference step (Beyer et al. 2016)1.
  • Crashes and unhandled exceptions. Error and crash telemetry captures the stack trace, request, and frequency of each exception thrown in production: a definite failure event, the errors oracle. Dedicated trackers (Sentry, Rollbar, Crashlytics) specialize in the grouping and alerting.
  • SLO violations. When a service-level objective is stated in measurable telemetry, monitoring reports whether the error budget is intact.
  • Runtime invariant violations. Structured events that carry invariants (cache hit ratio, queue depth, replication lag) let the dashboard catch drift a test suite cannot. When the invariant is a formal temporal property (an LTL or MTL spec such as "every acquire is eventually released"), a monitor compiled over the live event trace checks it continuously. This is runtime verification, the formal end of the same live invariant check.
  • Anomalies and unknown unknowns. A sudden shift in user-agent mix, traffic shape, or request size is hard to test for and easy to see on a dashboard; ad-hoc queries against rich telemetry answer questions no dashboard anticipated, the promise of observability over plain monitoring.

What monitoring does not catch is anything before the problem reaches production.

Infrastructure you operate: the SRE practice

For a service fleet the environment is uniform and reproducible, so the open question is what to instrument first.

Framework Metrics Layer it watches
USE (Gregg 2013)3 utilization, saturation, errors each resource: CPU, disk, network link, lock, queue
Four Golden Signals (Beyer et al. 2016)1 latency, traffic, errors, saturation the user-facing service (the SLO-level summary)
RED (Wilkie 2018)4 rate, errors, duration each request-driven service or endpoint (Golden Signals minus saturation)

RED is the request-path projection of the Golden Signals and USE the resource-path projection: RED on every service plus USE on every resource covers both, and the Golden Signals roll up to the user-facing SLO. Above the signals sit the objectives themselves, an SLO stated as a bound on a distribution and tracked as an error budget, and the health invariants (queue depth, replication lag, cache hit ratio) watched as live predicates.

Devices you don't control: field telemetry

A service runs on infrastructure the operator owns; field software runs where the operator cannot reach. The method is still observation of live signals, but the loss of control over the environment changes what that observation can do:

  • The telemetry is the only window. There is no second look: the operator cannot reproduce a user's exact device, OS build, locale, network path, or hardware, and cannot attach a debugger after the fact. Whatever the event captured is the entire record of a failure. For a service, monitoring is the last line behind everything reproducible upstream; for field software it is sometimes the only line, because the failing condition cannot be recreated in a lab.
  • The install base is the real input distribution. A service fleet is uniform by design; an install base is not. Field telemetry reports the device, OS-version, locale, screen-size, and connection-type mix that actually exists, the input distribution no test matrix enumerates up front. That distribution feeds back into what to test, making field telemetry a source of test inputs as much as a failure detector.
  • Collection is sampled, batched, and consent-bound. Events are sampled to control volume, queued and sent later over networks that drop them, and gated by user consent. Anonymous aggregate statistics are frequently the only lawful signal (opt-in telemetry, GDPR), so the oracle checks a distribution over a population rather than a single session's trace, and a late or missing event is the normal case, not an incident.
  • Crashes are harder to read. A field crash arrives without the context a server stack trace carries: it must be symbolicated against the right build, the install base spans many versions at once, and the payload is often a minidump rather than a live process.

The named signals differ per platform: browsers report Core Web Vitals and JavaScript errors through RUM; mobile reports crash-free-session rate, ANRs, and the store vitals (Play Console, App Store Connect); desktop reports crashes through Breakpad/Crashpad or Windows Error Reporting; IoT is the hard case, where intermittent connectivity forces store-and-forward and constrains how much can be sent at all. Product analytics (funnels, feature adoption, conversion) reuses the same pipelines but answers a different question and is out of scope here.

Tools

Metrics

  • Prometheus + Grafana — open-source de facto pair for pull-based metrics and dashboards.
  • Datadog, New Relic, Splunk Observability — managed metrics + APM; commercial.
  • VictoriaMetrics, Thanos, Mimir — long-term storage layers for Prometheus.
  • OpenTelemetry Metrics — vendor-neutral metrics API.

Logs

Tracing and APM

  • OpenTelemetry — the vendor-neutral standard; tracing, metrics, logs.
  • Jaeger, Zipkin, Tempo — open-source trace backends.
  • Datadog APM, New Relic, Dynatrace, AppDynamics — commercial.

Field telemetry (RUM, mobile, desktop, IoT)

  • Browser RUM: Sentry, Datadog RUM, New Relic Browser, Cloudflare Web Analytics. Real-user metrics for the page load and beyond.
  • Mobile: Firebase Crashlytics, Sentry, Embrace; crash-free rate, ANRs, and store vitals (Play Console, App Store Connect).
  • Desktop / native crash reporting: Google Breakpad / Crashpad, Windows Error Reporting, Mozilla crash-stats. Symbolicated crash and minidump collection from an uncontrolled install base.

Alerting and on-call

  • PagerDuty, Opsgenie, Grafana OnCall, Incident.io.
  • Alertmanager (Prometheus's alert router).

Continuous profiling

Always-on profilers in production; see profiling for the technique.

  • Pyroscope, Parca, Datadog Continuous Profiler, Google Cloud Profiler.

When to use, when not

Use:

  • Every production system. The minimum bar is Golden Signals (latency, traffic, errors, saturation) per service, alert on SLO violation, and a place to find logs.
  • Anywhere business-critical invariants need a runtime watcher (queue depth, replication lag, cache hit ratio, error-budget burn rate).
  • High-cardinality systems where the failure modes are unknown at design time. The observability discipline, wide events and rich labels, is what lets operators answer unforeseen questions.

Don't:

  • For pre-merge verification. Monitoring is downstream of every other method.
  • For low-volume internal tools where the operator is the user. The cost of running the stack outweighs the value.
  • Without an alerting strategy. Dashboards without alerts are monitoring theater; alerts without an on-call rotation are alert spam.
  • Without verifying the alerts themselves. An alarm threshold and an SLO definition are code, and usually the least-tested code in a system: an alarm nobody has fired on purpose is a claim about detection, not evidence of it. A synthetic breach shows it fires, and the alert history grouped by signature shows which definitions produce the pages.

Evidence

The empirical literature on monitoring is industrial reports: no controlled study compares monitored and unmonitored services, since monitoring is a baseline assumption across the literature rather than a treatment. The case for monitoring is definitional — a failure that produces no observable signal cannot be diagnosed or fixed.

Classification

  • Quality dimensions: Reliability, Performance, Functionality (when invariants fire).
  • Area: Production services, microservices, SaaS, on-call operations, and software running on devices the operator does not control (browsers, mobile, desktop, IoT) reporting field telemetry; anywhere live signals are observed against SLOs or field aggregates.
  • Guarantee: Heuristic.

Referenced by

References

  1. Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds. 2016. Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. https://sre.google/sre-book/. ↩↩↩

  2. Majors, Charity, Liz Fong-Jones, and George Miranda. 2022. Observability Engineering: Achieving Production Excellence. O'Reilly. ↩

  3. Gregg, Brendan. 2013. "Thinking Methodically about Performance." Communications of the ACM 56 (2): 45–51. https://doi.org/10.1145/2408776.2408791. ↩

  4. Wilkie, Tom. 2018. The RED Method: Patterns for Instrumentation and Monitoring. Talk, GrafanaCon EU. https://grafana.com/files/grafanacon_eu_2018/Tom_Wilkie_GrafanaCon_EU_2018.pdf. ↩