Skip to content

Software Quality

Load and stress testing

Load and stress testing drives a deployable system with synthetic traffic to measure its behavior under demand. The family covers four workloads, each answering a different question:

Workload Shape Question it answers Profile, and what to watch
Load Load profile: ramp up to the target level, hold steady, ramp downLoad profile: ramp up to the target level, hold steady, ramp down Can it handle the expected load? A representative profile at projected production volume; verify SLOs hold.
Stress Stress profile: load climbing in steps past capacity, then a vertical drop where the system breaksStress profile: load climbing in steps past capacity, then a vertical drop where the system breaks What breaks first when load exceeds capacity? Push past the design point until something fails; observe the failure mode.
Soak Soak profile: a moderate load held steady for a long timeSoak profile: a moderate load held steady for a long time Does anything degrade over long runs? Steady load for hours or days; watch for memory leaks, log-rotation issues, certificate expiry, FD exhaustion.
Spike Spike profile: a sudden brief jump to high load, then back to idleSpike profile: a sudden brief jump to high load, then back to idle What happens at the sudden onset of load? A step function of load (often to a level steady-state capacity could handle); check autoscaling lag, warm-up, and queue buildup.

The discipline is macro performance work; benchmarking (microbenchmarks of a function or an algorithm) is the micro counterpart, covered in microbenchmarking.

What it catches

  • SLO violations under target load. p99 latency above budget; error rate above budget; saturation of CPU, memory, or I/O.
  • Capacity miscalculations. Actual headroom is often less than design assumptions imply. The breaking point is N QPS, not 2N.
  • Tail-latency blowups. Median fine; p99 ten times worse than expected. Without explicit load testing, this surfaces only in production.
  • Queue and backpressure misbehavior. Unbounded queues that fill faster than the consumer drains; retry storms during partial failure; thundering-herd effects after a restart.
  • Resource leaks under sustained load. Memory creep, descriptor leaks, slow connection pools.
  • Autoscaling lag. Scale-up happens after the SLO is already breached.
  • Dependent-service collapse. The service under test stays within SLO while the database, message broker, or downstream API is the bottleneck.

Load testing does not by itself catch bugs in code paths the load profile leaves unexercised, nor real-traffic peculiarities the synthetic profile does not replicate (header mix, geographic distribution, attacker patterns). Composition with monitoring, canary deployment, and shadow-traffic patterns closes the gap.

Tools

Scriptable load generators

  • k6 — JavaScript-scripted; cloud or self-hosted; strong CI integration and built-in metrics output.
  • Locust — Python-scripted; user-class-and-task abstractions for realistic workloads.
  • Gatling — Scala / Java; protocol-rich.
  • Artillery — Node-scripted; YAML or JS test plans.
  • JMeter — Java; XML-configured, GUI-driven.

CLI-level traffic generators

  • wrk, wrk2 — high-throughput HTTP benchmarking against a single URL.
  • hey — Go-based HTTP load generator; minimal.
  • vegeta — Go-based; constant-rate workloads.
  • ab (Apache Bench) — basic HTTP benchmark.
  • fortio — Istio's load tool; supports HTTP/gRPC.

gRPC / protocol-specific

  • ghz — gRPC benchmarking.
  • k6 has gRPC support; Gatling has a gRPC plugin.

Database-targeted load

Capacity-planning and soak

  • Locust + Prometheus — long runs; metrics scraped while load generator emits.
  • k6 cloud / Grafana Cloud k6 — commercial managed services; useful for soak runs that exceed CI duration.

When to use, when not

Use:

  • Before any release that materially changes hot-path code or capacity assumptions.
  • For new services, before they take production traffic.
  • During capacity planning. The load test produces the measured capacity that goes into the projection; never use only design numbers.
  • For soak runs after long-uptime regressions. A six-hour run reveals leaks that a five-minute load test cannot.
  • For autoscaling and backpressure verification. Synthetic spikes exercise the scaling path on demand.

Don't:

  • Without a representative workload. Hitting one endpoint at peak rate doesn't model the real production mix; p99 under a synthetic profile is not p99 under production traffic.
  • Against production without explicit blast-radius control. Saturating staging is fine; saturating production is an outage unless paired with traffic shadowing and a stop button.
  • As the only performance method. Load testing shows that the system slowed; profiling shows why.
  • Once. Performance regressions ship continuously; a one-time pre-launch load test ages out within a release.

Evidence

Academic literature on load testing is thinner than on most other verification methods; the practice is industrial-dominant and benchmark-driven rather than study-driven. The case for it rests on the definitional point: nothing but load reveals how a system behaves under load.

Further reading

  • The Google SRE book — Handling Overload (Chapter 21) and the capacity-planning chapters cover the practice at industrial scale (Beyer et al. 2016)1.
  • YCSB — Cooper and colleagues define the cross-database comparison workloads (Cooper et al. 2010)2.

Performance

These aren't competing choices — they answer different performance questions, at different scopes. Microbenchmarking measures the cost of one small unit in isolation. Profiling explains where time or memory goes in a real workload. Load and stress testing drives the whole system under traffic to find where it degrades. You reach for whichever fits the question.

Classification

  • Quality dimensions: Performance, Reliability.
  • Area: Production web services, APIs, gRPC backends, databases; capacity planning, release gates, soak runs.
  • Guarantee: Empirical — the system survived the modeled profile; the next profile may surface different behavior.

Referenced by

References

  1. Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds. 2016. Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. https://sre.google/sre-book/. ↩

  2. Cooper, Brian F., Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. "Benchmarking Cloud Serving Systems with YCSB." Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC '10), 143–54. https://doi.org/10.1145/1807128.1807152. ↩