Latency, throughput, resource use under representative load. Most teams have monitoring; fewer have benchmarks in CI they trust.

What goes wrong

• Latency regressions on the hot path. A change increases p99 by 20 ms and nobody notices until customers do.
• Throughput collapse under load. The system handles 1× fine and falls over at 3×. Not visible at single-user load.
• Resource leaks. Memory, file descriptors, connection pools. Linear creep over hours or days; invisible in short tests.
• Accidental algorithmic complexity. $O(n^2)$ where $O(n \log n)$ was intended. Innocuous in tests with $n = 10$, catastrophic at $n = 10^6$.
• Coordination overhead at scale. Lock contention, cache misses, garbage-collection pauses. Manifest only above some throughput.
• Tail latency. p50 looks fine; p99.9 is a disaster. The mean is the wrong statistic for user-perceived latency.

Superlinear cost at scale

The everyday version is the database N+1 query: one query per row instead of one for the whole set, so cost climbs with the data. An empirical big-O regression gate catches it before production scale does, as does a profile of a realistic dataset. Tools that assert on the query count are in algorithmic complexity testing.

What success looks like

• Tail-latency percentiles are named and tracked. Specific percentiles tied to user impact (p99, p99.9), not the mean.
• SLOs and error budgets are explicit. "p99 < 200 ms on the search endpoint, error budget X minutes per quarter."
• Regressions are caught in CI, not in production. A microbenchmark or pre-merge load test fails before merge.
• The benchmark setup is reproducible. Same hardware, same data, same warm-up; numbers don't move by 30% between runs.

Referenced by

• Quality dimensions · Quality dimensions
• Choosing methods · Overview