# Microbenchmarking A microbenchmark measures the performance of one small unit: a function, an algorithm, a data-structure operation. It runs the unit many times and reports the result as a *distribution*, not a single number. The number a naive timer prints is meaningless on its own: warm-up effects, JIT compilation, CPU frequency scaling, cache state, and OS scheduling all move it run to run. A rigorous microbenchmark controls what it can (warm-up iterations, fixed input, pinned clocks), reports the machine it ran on, and *measures* what it cannot, reporting a mean or percentiles with a confidence interval so that "faster" is a defensible claim. When run on an uncontrolled host — thermal throttling, a noisy neighbor, a variable clock — it measures the environment, not the code. This is the *micro* counterpart to [load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md)'s *macro* work — one function in isolation, not the whole system under representative load. It measures a *single version*; finding which commit in a project's history made a benchmark regress is [change-point detection](https://quality.stereobooster.com/change-point-detection.md). ## What it catches - **Per-operation cost regressions.** A refactor that doubles the time of a hot function — caught against a recorded baseline, in CI, before it reaches a profile or a production dashboard. - **Algorithmic comparisons.** Which of two implementations is actually faster on the input that matters, with an interval that says whether the difference is real or noise. - **Empirical big-O.** Run the function across a sweep of input sizes and read the growth curve off the measured times — a quadratic that the code review missed shows up as a parabola. ## Tools - **CLI / language-agnostic:** [hyperfine](https://github.com/sharkdp/hyperfine) — benchmarks whole commands with warm-up runs, outlier detection, and statistical export. - **Rust:** [Criterion.rs](https://github.com/bheisler/criterion.rs) — collects a distribution, plots it, and reports a confidence interval and a regression verdict against the last run. - **JVM (Java/Kotlin/Scala):** [JMH](https://github.com/openjdk/jmh) — the OpenJDK harness built to defeat JIT and dead-code-elimination artifacts that wreck naive JVM timing. - **C++:** [Google Benchmark](https://github.com/google/benchmark) — repetition, statistics, and complexity (big-O) estimation. - **Python:** [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark) — fixture-driven timing with statistics, run inside the existing pytest suite. ## When to use, when not **Use:** - To make "faster" a defensible claim about a small unit. - To compare two implementations of the same function or algorithm on the input that matters. - To gate a hot path in CI against a recorded baseline. - To read empirical big-O off a sweep of input sizes. **Don't:** - For a verdict about the whole system. A function that benchmarks fast in isolation can be slow in situ under different cache state, contention, or input distribution; that question belongs to [profiling](https://quality.stereobooster.com/profiling.md) and [load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md). - For correctness. A microbenchmark measures cost, not whether the output is right; keep a [property](https://quality.stereobooster.com/property-based-testing.md) or [example](https://quality.stereobooster.com/example-tests.md) test alongside it. - On a shared CI runner, or any host you don't control. The remedy is to pin and isolate the machine, or to track drift over the project's history with [change-point detection](https://quality.stereobooster.com/change-point-detection.md) instead of asserting an absolute threshold on a noisy number. - Before profiling has shown the unit matters. Micro-optimizing a function that contributes a rounding error to the total spends effort where it cannot pay off. ## Evidence - **Statistically rigorous evaluation.** Georges, Buytaert, and Eeckhout showed that naive single-run benchmarking on a managed runtime produces conclusions that *reverse* under repeated measurement, and set out the confidence-interval methodology that tools like JMH and Criterion.rs now implement (Georges et al. 2007)[^georges2007]. The case is otherwise close to definitional: a noisy quantity needs a distributional measurement. ## Related **Performance** These aren't competing choices — they answer different performance questions, at different scopes. Microbenchmarking measures the cost of one small unit in isolation. [Profiling](https://quality.stereobooster.com/profiling.md) explains *where* time or memory goes in a real workload. [Load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md) drives the whole system under traffic to find where it degrades. You reach for whichever fits the question. ## Classification - **Quality dimensions:** Performance. - **Area:** Micro performance work in any language; comparing two implementations of a function or algorithm; hot-path cost measurement feeding a CI regression gate. - **Guarantee:** Heuristic — the benchmark reports *how this build performed on this input on this machine*, not how it will perform on every workload or every host. ## Referenced by - [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions - [Effect scope](https://quality.stereobooster.com/effect.md) · The axes - [Algorithmic complexity testing](https://quality.stereobooster.com/algorithmic-complexity.md) · Methods - [Change-point detection](https://quality.stereobooster.com/change-point-detection.md) · Methods - [Load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md) · Methods - [Statistical and sampling testing](https://quality.stereobooster.com/statistical-testing.md) · Methods - [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI - [Choosing methods](https://quality.stereobooster.com/choosing.md) · Overview ## References [^georges2007]: Georges, Andy, Dries Buytaert, and Lieven Eeckhout. 2007. "[Statistically Rigorous Java Performance Evaluation](https://www2.ccs.neu.edu/racket/Performance/andy-georges-paper.pdf)." *Proceedings of the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA '07)*, 57–76. .