Skip to content

Software Quality

Microbenchmarking

A microbenchmark measures the performance of one small unit: a function, an algorithm, a data-structure operation. It runs the unit many times and reports the result as a distribution, not a single number. The number a naive timer prints is meaningless on its own: warm-up effects, JIT compilation, CPU frequency scaling, cache state, and OS scheduling all move it run to run. A rigorous microbenchmark controls what it can (warm-up iterations, fixed input, pinned clocks), reports the machine it ran on, and measures what it cannot, reporting a mean or percentiles with a confidence interval so that "faster" is a defensible claim. When run on an uncontrolled host — thermal throttling, a noisy neighbor, a variable clock — it measures the environment, not the code.

This is the micro counterpart to load and stress testing's macro work — one function in isolation, not the whole system under representative load. It measures a single version; finding which commit in a project's history made a benchmark regress is change-point detection.

What it catches

  • Per-operation cost regressions. A refactor that doubles the time of a hot function — caught against a recorded baseline, in CI, before it reaches a profile or a production dashboard.
  • Algorithmic comparisons. Which of two implementations is actually faster on the input that matters, with an interval that says whether the difference is real or noise.
  • Empirical big-O. Run the function across a sweep of input sizes and read the growth curve off the measured times — a quadratic that the code review missed shows up as a parabola.

Tools

  • CLI / language-agnostic: hyperfine — benchmarks whole commands with warm-up runs, outlier detection, and statistical export.
  • Rust: Criterion.rs — collects a distribution, plots it, and reports a confidence interval and a regression verdict against the last run.
  • JVM (Java/Kotlin/Scala): JMH — the OpenJDK harness built to defeat JIT and dead-code-elimination artifacts that wreck naive JVM timing.
  • C++: Google Benchmark — repetition, statistics, and complexity (big-O) estimation.
  • Python: pytest-benchmark — fixture-driven timing with statistics, run inside the existing pytest suite.

When to use, when not

Use:

  • To make "faster" a defensible claim about a small unit.
  • To compare two implementations of the same function or algorithm on the input that matters.
  • To gate a hot path in CI against a recorded baseline.
  • To read empirical big-O off a sweep of input sizes.

Don't:

  • For a verdict about the whole system. A function that benchmarks fast in isolation can be slow in situ under different cache state, contention, or input distribution; that question belongs to profiling and load and stress testing.
  • For correctness. A microbenchmark measures cost, not whether the output is right; keep a property or example test alongside it.
  • On a shared CI runner, or any host you don't control. The remedy is to pin and isolate the machine, or to track drift over the project's history with change-point detection instead of asserting an absolute threshold on a noisy number.
  • Before profiling has shown the unit matters. Micro-optimizing a function that contributes a rounding error to the total spends effort where it cannot pay off.

Evidence

  • Statistically rigorous evaluation. Georges, Buytaert, and Eeckhout showed that naive single-run benchmarking on a managed runtime produces conclusions that reverse under repeated measurement, and set out the confidence-interval methodology that tools like JMH and Criterion.rs now implement (Georges et al. 2007)1.

The case is otherwise close to definitional: a noisy quantity needs a distributional measurement.

Performance

These aren't competing choices — they answer different performance questions, at different scopes. Microbenchmarking measures the cost of one small unit in isolation. Profiling explains where time or memory goes in a real workload. Load and stress testing drives the whole system under traffic to find where it degrades. You reach for whichever fits the question.

Classification

  • Quality dimensions: Performance.
  • Area: Micro performance work in any language; comparing two implementations of a function or algorithm; hot-path cost measurement feeding a CI regression gate.
  • Guarantee: Heuristic — the benchmark reports how this build performed on this input on this machine, not how it will perform on every workload or every host.

Referenced by

References

  1. Georges, Andy, Dries Buytaert, and Lieven Eeckhout. 2007. "Statistically Rigorous Java Performance Evaluation." Proceedings of the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA '07), 57–76. https://doi.org/10.1145/1297105.1297033. ↩