Skip to content

Software Quality

Flaky tests

A flaky test passes and fails on the same code, so a red run may mean a defect or may mean nothing, and telling which requires a human to look. Behind it is a source of nondeterminism nobody accounted for.

What makes tests flaky

The causes group by where the nondeterminism sits (Parry et al. 2021)1:

  • Within one test — concurrency, randomness, floating-point comparison, the iteration order of an unordered collection, an assertion range narrower than the valid output, and a per-test timeout.
  • Between tests — order dependency, a leaked resource the next test inherits, and a suite-wide timeout.
  • Outside the process — an asynchronous call the test does not wait for, I/O, the network, system time, and platform differences.

Provoking flakiness on purpose

A flakiness detector varies something that must not affect the outcome, then diffs the verdicts against an unvaried run.

Randomize the order

A test is order-dependent when some reordered subsequence of the suite changes its pass/fail result from the result it gives in the default order (Zhang et al. 2014)2. Running the suite shuffled and diffing against the default run finds them.

Most runners ship the shuffle: pytest-randomly and pytest-random-order, go test -shuffle=on, RSpec's --order random, JUnit 5's ClassOrderer/MethodOrderer, Vitest's sequence.shuffle, and Maven's Surefire plugin with -Dsurefire.runOrder=random. Without a logged seed a failure cannot be reproduced.

Vary the environment

96% of flaky tests are independent of the platform, so environment dependence takes priority over platform dependence (Luo et al. 2014)3. What to vary: TZ, locale and collation, filesystem encoding, the number of visible CPUs, hostname, HOME, and the working directory. On the JVM, NonDex varies the iteration order of APIs whose specification does not fix one, so a test that silently assumed HashMap ordering fails.

Add resource contention

Timing-sensitive tests fail when the machine is busy, which is why they fail in CI and pass locally. Making the machine busy on purpose turns that into a detector: Shaker, which runs the suite alongside competing CPU and memory load, reports finding more flaky tests with fewer executions than rerunning unchanged (Silva et al. 2021)4. The same competition comes from constraining the container's CPU quota or running a load generator beside the suite.

Rerun, and what that costs

Rerunning until the verdict changes is the weakest detector per unit of compute: reaching 95% confidence that a passing test is not flaky for a non-order-dependent reason would take at least 170 reruns (Gruber et al. 2021)5. A budget of two or three retries is therefore a retry policy rather than a detector. go test -count=N does the work.

When to look

Flakiness is mostly present from the start, so the cheapest moment to provoke a new test is the commit introducing it. Provoking only newly added tests detects 75% of flaky tests, rising to 85% when directly modified tests are provoked too (Lam et al. 2020)6.

What to do with a flaky test

Repair beats deletion: 24% of flaky-test fixes changed the code under test as well as the test, and 94% of those fixed a real bug in it (Luo et al. 2014)3. The repair addresses the source rather than the symptom: waiting on a condition instead of a delay, confining the clock read so the test cannot reach the system clock, removing the shared static, or making the fixture rebuild the state it depends on.

Quarantine stops a test from gating the pipeline; retry re-runs it until it passes. Neither repairs the test.

Referenced by

References

  1. Parry, Owain, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. 2021. "A Survey of Flaky Tests." ACM Transactions on Software Engineering and Methodology 31 (1): 1–74. https://doi.org/10.1145/3476105. ↩

  2. Zhang, Sai, Darioush Jalali, Jochen Wuttke, et al. 2014. "Empirically Revisiting the Test Independence Assumption." International Symposium on Software Testing and Analysis (ISSTA), 385–96. https://doi.org/10.1145/2610384.2610404. ↩

  3. Luo, Qingzhou, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. "An Empirical Analysis of Flaky Tests." Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '14) (Hong Kong, China), 643–53. https://doi.org/10.1145/2635868.2635920. ↩↩

  4. Silva, Denini, Leopoldo Teixeira, and Marcelo d'Amorim. 2021. "Shaker: A Tool for Detecting More Flaky Tests Faster." IEEE/ACM International Conference on Automated Software Engineering (ASE), 1281–85. https://doi.org/10.1109/ASE51524.2021.9678918. ↩

  5. Gruber, Martin, Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2021. "An Empirical Study of Flaky Tests in Python." IEEE Conference on Software Testing, Verification and Validation (ICST), 148–58. https://doi.org/10.1109/ICST49551.2021.00026. ↩

  6. Lam, Wing, Stefan Winter, Anjiang Wei, Tao Xie, Darko Marinov, and Jonathan Bell. 2020. "A Large-Scale Longitudinal Study of Flaky Tests." Proceedings of the ACM on Programming Languages 4 (OOPSLA): 1–29. https://doi.org/10.1145/3428270. ↩