# Flaky tests A **flaky** test passes and fails on the same code, so a red run may mean a defect or may mean nothing, and telling which requires a human to look. Behind it is a source of nondeterminism nobody accounted for. ## What makes tests flaky The causes group by where the nondeterminism sits (Parry et al. 2021)[^parry2021]: - **Within one test** — concurrency, randomness, floating-point comparison, the iteration order of an unordered collection, an assertion range narrower than the valid output, and a per-test timeout. - **Between tests** — order dependency, a leaked resource the next test inherits, and a suite-wide timeout. - **Outside the process** — an asynchronous call the test does not wait for, I/O, the network, system time, and platform differences. ## Provoking flakiness on purpose A flakiness detector varies something that must not affect the outcome, then diffs the verdicts against an unvaried run. ### Randomize the order A test is *order-dependent* when some reordered subsequence of the suite changes its pass/fail result from the result it gives in the default order (Zhang et al. 2014)[^zhang2014]. Running the suite shuffled and diffing against the default run finds them. Most runners ship the shuffle: [pytest-randomly](https://github.com/pytest-dev/pytest-randomly) and [pytest-random-order](https://github.com/pytest-dev/pytest-random-order), `go test -shuffle=on`, [RSpec](https://rspec.info/)'s `--order random`, [JUnit 5](https://junit.org/junit5/)'s `ClassOrderer`/`MethodOrderer`, [Vitest](https://vitest.dev/)'s `sequence.shuffle`, and Maven's [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) plugin with `-Dsurefire.runOrder=random`. Without a logged seed a failure cannot be reproduced. ### Vary the environment **96% of flaky tests are independent of the platform**, so environment dependence takes priority over platform dependence (Luo et al. 2014)[^luo2014]. What to vary: `TZ`, locale and collation, filesystem encoding, the number of visible CPUs, hostname, `HOME`, and the working directory. On the JVM, [NonDex](https://github.com/TestingResearchIllinois/NonDex) varies the iteration order of APIs whose specification does not fix one, so a test that silently assumed `HashMap` ordering fails. ### Add resource contention Timing-sensitive tests fail when the machine is busy, which is why they fail in CI and pass locally. Making the machine busy on purpose turns that into a detector: Shaker, which runs the suite alongside competing CPU and memory load, reports finding more flaky tests with fewer executions than rerunning unchanged (Silva et al. 2021)[^silva2021]. The same competition comes from constraining the container's CPU quota or running a load generator beside the suite. ### Rerun, and what that costs Rerunning until the verdict changes is the weakest detector per unit of compute: reaching 95% confidence that a passing test is *not* flaky for a non-order-dependent reason would take at least **170 reruns** (Gruber et al. 2021)[^gruber2021]. A budget of two or three retries is therefore a retry policy rather than a detector. `go test -count=N` does the work. ## When to look Flakiness is mostly present from the start, so the cheapest moment to provoke a new test is the commit introducing it. Provoking only newly added tests detects 75% of flaky tests, rising to 85% when directly modified tests are provoked too (Lam et al. 2020)[^lam2020]. ## What to do with a flaky test Repair beats deletion: 24% of flaky-test fixes changed the code under test as well as the test, and 94% of those fixed a real bug in it (Luo et al. 2014)[^luo2014]. The repair addresses the source rather than the symptom: waiting on a condition instead of a delay, confining the clock read so the test cannot reach the system clock, removing the shared static, or making the fixture rebuild the state it depends on. Quarantine stops a test from gating the pipeline; retry re-runs it until it passes. Neither repairs the test. ## Referenced by - [Maintainability](https://quality.stereobooster.com/maintainability.md) · Quality dimensions - [Effect scope](https://quality.stereobooster.com/effect.md) · The axes - [Guarantee](https://quality.stereobooster.com/guarantee.md) · The axes - [Measuring test-suite effectiveness](https://quality.stereobooster.com/measuring-test-effectiveness.md) · Methods - [The test suite as an object](https://quality.stereobooster.com/test-suite.md) · Methods - [Verifying time and date handling](https://quality.stereobooster.com/time-and-date.md) · Methods ## References [^parry2021]: Parry, Owain, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. 2021. "[A Survey of Flaky Tests](https://o-parry.github.io/papers/2021a.pdf)." *ACM Transactions on Software Engineering and Methodology* 31 (1): 1–74. . [^zhang2014]: Zhang, Sai, Darioush Jalali, Jochen Wuttke, et al. 2014. "[Empirically Revisiting the Test Independence Assumption](https://zhang-sai.github.io/pdf/zhang-issta14.pdf)." *International Symposium on Software Testing and Analysis (ISSTA)*, 385–96. . [^luo2014]: Luo, Qingzhou, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. "[An Empirical Analysis of Flaky Tests](https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf)." *Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '14)* (Hong Kong, China), 643–53. . [^silva2021]: Silva, Denini, Leopoldo Teixeira, and Marcelo d'Amorim. 2021. "[Shaker: A Tool for Detecting More Flaky Tests Faster](https://damorim.github.io/publications/Cordeiro_ETAL_ASETD2021.pdf)." *IEEE/ACM International Conference on Automated Software Engineering (ASE)*, 1281–85. . [^gruber2021]: Gruber, Martin, Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2021. "[An Empirical Study of Flaky Tests in Python](https://arxiv.org/pdf/2101.09077)." *IEEE Conference on Software Testing, Verification and Validation (ICST)*, 148–58. . [^lam2020]: Lam, Wing, Stefan Winter, Anjiang Wei, Tao Xie, Darko Marinov, and Jonathan Bell. 2020. "[A Large-Scale Longitudinal Study of Flaky Tests](https://cs.stanford.edu/~anjiang/papers/LamETAL20LongitudinalFlakyTests.pdf)." *Proceedings of the ACM on Programming Languages* 4 (OOPSLA): 1–29. .