Skip to content

Software Quality

Automated test generation

Automated test generation builds a runnable test suite without a developer writing the cases by hand. Two families share the goal and the output — a persisted suite of tests with assertions — and differ only in how they pick inputs. Search-based generators (EvoSuite, Pynguin) run an evolutionary search steered by a coverage-distance fitness function; this is the search-based paradigm applied to structured tests. Feedback-directed random generators (Randoop) build sequences incrementally, extending only those that ran without error. That construction is cheaper and simpler than a fitness-guided search, and weaker on hard-to-reach branches. Both target object-oriented code, where a candidate test is a sequence of statements that constructs objects, calls methods on them, and passes the results as arguments to further calls.

Because a test's input already is a program fragment, persisting it means emitting source: these tools write JUnit or pytest files a developer keeps in the repo. That code output separates these tools from a fuzzer, which emits a crashing input at runtime rather than a reusable test.

Model-based testing is a different sense of "generate tests": it drives an implementation through a state model rather than building tests from the code itself, and it is covered under state machines.

What it catches

  • Coverage. A generator reaches high branch coverage on code that had no tests at all.
  • A regression baseline. Assertions that pin the code's current observed behavior, so a later change that alters it fails a test.
  • Crashes and uncaught exceptions. The search flags any call sequence that makes the code throw where it should not, needing no oracle beyond the exception itself.

What it does not catch is incorrect behavior that does not crash: with no specification to check against, the generated oracle treats whatever the code does today as correct.

Whole-suite and many-objective generation

Early search-based generation optimized one coverage target at a time, which wastes budget: easy branches get covered over and over while a hard one starves. Whole-suite generation (Fraser and Arcuri 2013)1, introduced by EvoSuite, evolves an entire test suite against all coverage goals at once, so the search allocates effort where it is still needed.

The current formulation is many-objective. MOSA (Panichella et al. 2015)2 treats each branch as its own optimization objective and uses a preference-sorting genetic algorithm to keep, for every uncovered branch, the test that comes closest to covering it. DynaMOSA (Panichella et al. 2018)3 adds dynamic target selection: it optimizes only those branches whose control-dependency prerequisites are already satisfied, expanding the set of active targets as the frontier advances. DynaMOSA is the default algorithm in EvoSuite and generally the strongest on coverage.

The oracle problem

A generator supplies inputs and can find crashes on its own, but it cannot tell what the code is supposed to do. EvoSuite closes the gap with a regression oracle: it runs the generated tests, observes the return values and object state the code currently produces, and writes assertions that pin exactly those. To keep only the assertions that would notice a change, it applies mutation analysis, retaining assertions that kill mutants and dropping the rest.

The consequence is the method's central limitation: the generated suite is a change detector, not a correctness checker. It asserts the code's present behavior, bugs and all, as if it were the specification. A generator pointed at a function with an off-by-one error faithfully asserts the wrong answer, then fails the day someone fixes it.

Tools

  • EvoSuite (Java) is the reference implementation (Fraser and Arcuri 2011)4 and the origin of whole-suite and DynaMOSA generation. It produces JUnit tests with regression assertions and mutation-filtered oracles.
  • Pynguin (Lukasczyk and Fraser 2022)5 ports MOSA and DynaMOSA to Python. Dynamic typing makes the problem harder, since the generator cannot read parameter types to know what to pass, so it leans on type hints and runtime feedback.
  • Randoop (Java) is the feedback-directed random generator. Its JUnit output comes in two parts: error-revealing tests for the sequences that failed, and regression tests whose assertions capture current behavior.
  • CODAMOSA (Lemieux et al. 2023)6 is an LLM hybrid: when DynaMOSA's coverage stops improving (a plateau), it asks a language model for fresh test cases to jump to unexplored regions, then resumes the search from there.

When to use, when not

Use:

  • Bootstrapping a regression suite on untested legacy code. A generator produces a high-coverage suite that pins current behavior, a safety net for refactoring code that has no tests.
  • Filling coverage gaps a hand-written suite left, as a generator of candidate cases a developer then reviews and re-purposes.
  • Crash and uncaught-exception hunting, where the implicit oracle (the code threw where it should not) needs no specification.

Don't:

  • As a source of correctness oracles. The regression oracle cannot tell right behavior from wrong; it encodes whatever the code does now. Establishing that the code is correct, not merely unchanged, takes a real oracle: a specification, invariants, or a reference implementation.
  • When readable, intent-revealing tests matter more than coverage. Generated tests document behavior poorly and are hard to maintain.

Evidence

  • Coverage rises with the search formulation. Optimizing a whole suite against all branches at once covered more branches than targeting one goal at a time, with smaller suites, across 1,741 Java classes (Fraser and Arcuri 2013)1; DynaMOSA's dynamic target selection covered more branches still than the whole-suite archive variant, on 28% of 346 classes (Panichella et al. 2018)3.
  • Coverage attained is not faults found. Suites generated by EvoSuite, Randoop, and a third tool detected 55.7% of the 357 real faults in Defects4J between them, and no single tool more than 40.6%; of the faults none of them detected, 63.3% had been covered by a generated test that did not reveal them (Shamshiri et al. 2015)7.
  • Generated suites did not help testers find more faults. Across two controlled experiments with 49 and 48 subjects, those given EvoSuite reached higher coverage than those testing by hand, found the same number of seeded faults, and produced suites that did not always score higher under mutation analysis (Fraser et al. 2015)8.
  • Scope. Automated generation is strong at the coverage and regression-detection job and weak at the correctness job. It supplies inputs and a change-detecting oracle; it does not supply a specification, and no amount of search invents one.

Generative testing

These methods blur together because they all run the code and check the result without a hand-written expected value. Two roles pull them apart, and a single test picks one of each.

Input generators choose the input, and differ by the steering signal (the Generative subtree of the input axis):

  • Random: property-based testing samples from a generator or schema.
  • Coverage feedback: fuzzing mutates inputs steered by coverage (raw bytes); automated test generation runs the same feedback loop as a fitness-guided search over structured call sequences.
  • Solver: symbolic execution derives an input that reaches a chosen path.
  • Systematic: combinatorial testing builds a covering array over every t-way combination.

Oracle suppliers provide the verdict when no expected value is written:

Pick one generator and one oracle: they compose. A coverage-guided fuzzer that checks a metamorphic relation is fuzzing and metamorphic at once. (The written-answer end, where the author picks rows and answers by hand, is example / parameterized tests.)

Classification

  • Quality dimensions: Functionality, Reliability (uncaught-exception and crash discovery during generation).
  • Area: Example test generation (Java, Python); raising coverage on legacy code with no tests; producing a regression suite that pins current behavior.
  • Guarantee: Heuristic.

Referenced by

References

  1. Fraser, Gordon, and Andrea Arcuri. 2013. "Whole Test Suite Generation." IEEE Transactions on Software Engineering 39 (2): 276–91. https://doi.org/10.1109/TSE.2012.14. ↩↩

  2. Panichella, Annibale, Fitsum Meshesha Kifetew, and Paolo Tonella. 2015. "Reformulating Branch Coverage as a Many-Objective Optimization Problem." 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), 1–10. https://doi.org/10.1109/ICST.2015.7102604. ↩

  3. Panichella, Annibale, Fitsum Meshesha Kifetew, and Paolo Tonella. 2018. "Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets." IEEE Transactions on Software Engineering 44 (2): 122–58. https://doi.org/10.1109/TSE.2017.2663435. ↩↩

  4. Fraser, Gordon, and Andrea Arcuri. 2011. "EvoSuite: Automatic Test Suite Generation for Object-Oriented Software." Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE '11), 416–19. https://doi.org/10.1145/2025113.2025179. ↩

  5. Lukasczyk, Stephan, and Gordon Fraser. 2022. "Pynguin: Automated Unit Test Generation for Python." Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion '22), 168–72. https://doi.org/10.1145/3510454.3516829. ↩

  6. Lemieux, Caroline, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. "CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models." 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 919–31. https://doi.org/10.1109/ICSE48619.2023.00085. ↩

  7. Shamshiri, Sina, René Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. "Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges." 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 201–11. https://doi.org/10.1109/ASE.2015.86. ↩

  8. Fraser, Gordon, Matt Staats, Phil McMinn, Andrea Arcuri, and Frank Padberg. 2015. "Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study." ACM Transactions on Software Engineering and Methodology 24 (4): 23:1–49. https://doi.org/10.1145/2699688. ↩