Skip to content

Software Quality

Load and stress testing

Load and stress testing drives a deployable system with synthetic traffic to measure its behavior under demand. The family covers four workloads, each answering a different question:

Workload Shape Question it answers Profile, and what to watch
Load Load profile: ramp up to the target level, hold steady, ramp downLoad profile: ramp up to the target level, hold steady, ramp down Can it handle the expected load? A representative profile at projected production volume; verify SLOs hold.
Stress Stress profile: load climbing in steps past capacity, then a vertical drop where the system breaksStress profile: load climbing in steps past capacity, then a vertical drop where the system breaks What breaks first when load exceeds capacity? Push past the design point until something fails; observe the failure mode.
Soak Soak profile: a moderate load held steady for a long timeSoak profile: a moderate load held steady for a long time Does anything degrade over long runs? Steady load for hours or days; watch for memory leaks, log-rotation issues, certificate expiry, FD exhaustion.
Spike Spike profile: a sudden brief jump to high load, then back to idleSpike profile: a sudden brief jump to high load, then back to idle What happens at the sudden onset of load? A step function of load (often to a level steady-state capacity could handle); check autoscaling lag, warm-up, and queue buildup.

The discipline is macro performance work; benchmarking (microbenchmarks of a function or an algorithm) is the micro counterpart, covered in microbenchmarking.

What it catches

  • SLO violations under target load. p99 latency above budget; error rate above budget; saturation of CPU, memory, or I/O.
  • Capacity miscalculations. Actual headroom is often less than design assumptions imply. The breaking point is N QPS, not 2N.
  • Tail-latency blowups. Median fine; p99 ten times worse than expected. Without explicit load testing, this surfaces only in production.
  • Queue and backpressure misbehavior. Unbounded queues that fill faster than the consumer drains; retry storms during partial failure; thundering-herd effects after a restart.
  • Resource leaks under sustained load. Memory creep, descriptor leaks, slow connection pools.
  • Autoscaling lag. Scale-up happens after the SLO is already breached.
  • Dependent-service collapse. The service under test stays within SLO while the database, message broker, or downstream API is the bottleneck.

Load testing does not by itself catch bugs in code paths the load profile leaves unexercised, nor real-traffic peculiarities the synthetic profile does not replicate (header mix, geographic distribution, attacker patterns). Composition with monitoring, canary deployment, and shadow-traffic patterns closes the gap.

Tools

Scriptable load generators

  • k6 — JavaScript-scripted; cloud or self-hosted; strong CI integration and built-in metrics output.
  • Locust — Python-scripted; user-class-and-task abstractions for realistic workloads.
  • Gatling — Scala / Java; protocol-rich.
  • Artillery — Node-scripted; YAML or JS test plans.
  • JMeter — Java; XML-configured, GUI-driven.

CLI-level traffic generators

  • wrk — high-throughput HTTP benchmarking against a single URL.
  • wrk2 — a wrk variant that holds a constant request rate and records latency without coordinated omission.
  • hey — Go-based HTTP load generator; minimal.
  • vegeta — Go-based; constant-rate workloads.
  • ab (Apache Bench) — basic HTTP benchmark.
  • fortio — Istio's load tool; supports HTTP/gRPC.

gRPC / protocol-specific

  • ghz — gRPC benchmarking.
  • k6 has gRPC support.
  • Gatling has a gRPC plugin.

Database-targeted load

  • pgbench — ships with PostgreSQL; runs a simple bank-transfer transaction mix by default.
  • sysbench — Lua-scripted benchmark for MySQL and PostgreSQL, and for CPU, memory and file I/O.
  • YCSB — the Yahoo Cloud Serving Benchmark, a key-value workload suite run against many databases.
  • TPC-C / TPC-H — standard transaction-processing and decision-support workload specifications.

Capacity-planning and soak

  • Locust + Prometheus — long runs; metrics scraped while the load generator emits.
  • Grafana Cloud k6 — a commercial managed service; useful for soak runs that exceed CI duration.

When to use, when not

Use:

  • Before any release that materially changes hot-path code or capacity assumptions.
  • For new services, before they take production traffic.
  • During capacity planning. The load test produces the measured capacity that goes into the projection in place of design numbers.
  • For soak runs after long-uptime regressions. A six-hour run reveals leaks that a five-minute load test cannot.
  • For autoscaling and backpressure verification. Synthetic spikes exercise the scaling path on demand.

Don't:

  • Without a representative workload. Hitting one endpoint at peak rate doesn't model the real production mix; p99 under a synthetic profile is not p99 under production traffic.
  • Against production without explicit blast-radius control. Saturating staging is fine; saturating production is an outage unless paired with traffic shadowing and a stop button.
  • As the only performance method. Load testing shows that the system slowed; profiling shows why.
  • Once. Performance regressions ship continuously; a one-time pre-launch load test ages out within a release.

Evidence

The support for load testing is definitional, not statistical: nothing but load reveals how a system behaves under load. Which way of designing that load works best is unmeasured: Jiang and Hassan's survey of load-testing research and practice found the effectiveness of the load-design techniques it reviewed, in scale and coverage, not established, and almost no metric for what a load covers beyond aggregate workload (Jiang and Hassan 2015)1.

Further reading

  • The Google SRE book — its introduction makes regular load testing a mandatory step of capacity planning (Beyer et al. 2016)2; Handling Overload (Chapter 21) covers what a serving system does once load exceeds capacity: degraded responses, client-side throttling, and retry limits (Forero Cuervo 2016)3.
  • YCSB — Cooper and colleagues define the cross-database comparison workloads (Cooper et al. 2010)4.

Performance

These aren't competing choices — they answer different performance questions, at different scopes. Microbenchmarking measures the cost of one small unit in isolation. Profiling explains where time or memory goes in a real workload. Load and stress testing drives the whole system under traffic to find where it degrades. You reach for whichever fits the question.

Classification

  • Quality dimensions: Performance, Reliability.
  • Area: Production web services, APIs, gRPC backends, databases; capacity planning, release gates, soak runs.
  • Guarantee: Empirical — the system survived the modeled profile; the next profile may surface different behavior.

Referenced by

References

  1. Jiang, Zhen Ming, and Ahmed E. Hassan. 2015. "A Survey on Load Testing of Large-Scale Software Systems." IEEE Transactions on Software Engineering 41 (11): 1091–118. https://doi.org/10.1109/TSE.2015.2445340. ↩

  2. Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds. 2016. Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. https://sre.google/sre-book/. ↩

  3. Forero Cuervo, Alejandro. 2016. "Handling Overload." In Site Reliability Engineering: How Google Runs Production Systems, edited by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. O'Reilly. https://sre.google/sre-book/handling-overload/. ↩

  4. Cooper, Brian F., Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. "Benchmarking Cloud Serving Systems with YCSB." Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC '10), 143–54. https://doi.org/10.1145/1807128.1807152. ↩