Skip to content

Software Quality

Automated test generation

Automated test generation builds a runnable test suite without a developer writing the cases by hand. Two families share the goal and the output — a persisted suite of tests with assertions — and differ only in how they pick inputs. Search-based generators (EvoSuite, Pynguin) run an evolutionary search steered by a coverage-distance fitness function; this is the search-based paradigm applied to structured tests. Feedback-directed random generators (Randoop) build sequences incrementally, extending only those that ran without error, rather than running a fitness-guided search: cheaper and simpler, weaker on hard-to-reach branches. Both target object-oriented code, where a candidate test is a sequence of statements: construct objects, call methods on them, pass the results as arguments to further calls.

Because a test's input already is a program fragment, persisting it means emitting source: these tools write JUnit or pytest files a developer keeps in the repo. That code output is the defining trait against a fuzzer, which emits a crashing input at runtime rather than a reusable test.

A different sense of "generate tests" drives an implementation through a state model rather than building tests from the code itself; that is model-based testing, covered under state machines.

What it catches

  • Coverage. A high-branch-coverage suite over code that had none, generated in minutes rather than written by hand.
  • A regression baseline. Assertions that pin the code's current observed behavior, so a later change that alters it fails a test.
  • Crashes and uncaught exceptions. The search flags any call sequence that makes the code throw where it should not, needing no oracle beyond the exception itself.

What it does not catch is incorrect behavior that does not crash: with no specification to check against, the generated oracle treats whatever the code does today as correct.

Whole-suite and many-objective generation

Early search-based generation optimized one coverage target at a time, which wastes budget: easy branches get covered over and over while a hard one starves. Whole-suite generation (Fraser and Arcuri 2013)1, introduced by EvoSuite, evolves an entire test suite against all coverage goals at once, so the search allocates effort where it is still needed.

The current formulation is many-objective. MOSA (Panichella et al. 2015)2 treats each branch as its own optimization objective and uses a preference-sorting genetic algorithm to keep, for every uncovered branch, the test that comes closest to covering it. DynaMOSA (Panichella et al. 2018)3 adds dynamic target selection: it optimizes only those branches whose control-dependency prerequisites are already satisfied, expanding the set of active targets as the frontier advances. DynaMOSA is the default algorithm in EvoSuite and generally the strongest on coverage.

The oracle problem

A generator supplies inputs and can find crashes on its own, but it cannot tell what the code is supposed to do. EvoSuite closes the gap with a regression oracle: it runs the generated tests, observes the return values and object state the code currently produces, and writes assertions that pin exactly those. To keep only the assertions that would notice a change, it applies mutation analysis, retaining assertions that kill mutants and dropping the rest.

The consequence is the method's central limitation: the generated suite is a change-detector, not a correctness-checker. It asserts the code's present behavior, bugs and all, as if it were the specification. A generator pointed at a function with an off-by-one error faithfully asserts the wrong answer, then fails the day someone fixes it.

Tools

  • EvoSuite (Java) is the reference implementation (Fraser and Arcuri 2011)4 and the origin of whole-suite and DynaMOSA generation. It produces JUnit tests with regression assertions and mutation-filtered oracles.
  • Pynguin (Lukasczyk and Fraser 2022)5 ports MOSA and DynaMOSA to Python. Dynamic typing makes the problem harder, since the generator cannot read parameter types to know what to pass, so it leans on type hints and runtime feedback.
  • Randoop (Java) is the feedback-directed random generator. Its JUnit output comes in two parts: error-revealing tests for the sequences that failed, and regression tests whose assertions capture current behavior.
  • CODAMOSA (Lemieux et al. 2023)6 is an LLM hybrid: when DynaMOSA's coverage stops improving (a plateau), it asks a language model for fresh test cases to jump to unexplored regions, then resumes the search from there.

When to use, when not

Use:

  • Bootstrapping a regression suite on untested legacy code. A generator produces a high-coverage suite that pins current behavior, a safety net for refactoring code that has no tests.
  • Filling coverage gaps a hand-written suite left, as a generator of candidate cases a developer then reviews and re-purposes.
  • Crash and uncaught-exception hunting, where the implicit oracle (the code threw where it should not) needs no specification.

Don't:

  • As a source of correctness oracles. The regression oracle cannot tell right behavior from wrong; it encodes whatever the code does now. Establishing that the code is correct, not merely unchanged, takes a real oracle: a specification, invariants, or a reference implementation.
  • When readable, intent-revealing tests matter more than coverage. Generated tests document behavior poorly and are hard to maintain.

Evidence

  • Automated generation attains high coverage cheaply, replicated across the tool competitions and the whole-suite and many-objective studies (Fraser and Arcuri 2013; Panichella et al. 2018)1 3.
  • High coverage does not translate into finding real faults. Shamshiri and colleagues ran EvoSuite, Randoop, and a third generator against a curated set of real bugs and found the generated suites detected only a minority of them, despite high coverage, because the regression oracle cannot flag behavior it was trained to accept (Shamshiri et al. 2015)7.
  • Generated tests did not clearly help developers. In a controlled study, testers given EvoSuite suites did not reliably outperform those writing tests by hand: the machine-generated assertions were hard to read, and pinning current (possibly buggy) behavior as expected misled as often as it helped (Fraser et al. 2015)8.
  • Scope. Automated generation is strong at the coverage and regression-detection job and weak at the correctness job. It supplies inputs and a change-detecting oracle; it does not supply a specification, and no amount of search invents one.

Generative testing

These methods blur together because they all run the code and check the result without a hand-written expected value. Two roles pull them apart, and a single test picks one of each.

Input generators choose the input, and differ by the steering signal (the Generative subtree of the input axis):

  • Random: property-based testing samples from a generator or schema.
  • Coverage feedback: fuzzing mutates inputs steered by coverage (raw bytes); automated test generation runs the same feedback loop as a fitness-guided search over structured call sequences.
  • Solver: symbolic execution derives an input that reaches a chosen path.
  • Systematic: combinatorial testing builds a covering array over every t-way combination.

Oracle suppliers provide the verdict when no expected value is written:

Pick one generator and one oracle: they compose. A coverage-guided fuzzer that checks a metamorphic relation is fuzzing and metamorphic at once. (The written-answer end, where the author picks rows and answers by hand, is example / parameterized tests.)

Classification

  • Quality dimensions: Functionality, Reliability (uncaught-exception and crash discovery during generation).
  • Area: Example test generation (Java, Python); raising coverage on legacy code with no tests; producing a regression suite that pins current behavior.
  • Guarantee: Heuristic.

Referenced by

References

  1. Fraser, Gordon, and Andrea Arcuri. 2013. "Whole Test Suite Generation." IEEE Transactions on Software Engineering 39 (2): 276–91. https://doi.org/10.1109/TSE.2012.14. ↩↩

  2. Panichella, Annibale, Fitsum Meshesha Kifetew, and Paolo Tonella. 2015. "Reformulating Branch Coverage as a Many-Objective Optimization Problem." 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), 1–10. https://doi.org/10.1109/ICST.2015.7102604. ↩

  3. Panichella, Annibale, Fitsum Meshesha Kifetew, and Paolo Tonella. 2018. "Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets." IEEE Transactions on Software Engineering 44 (2): 122–58. https://doi.org/10.1109/TSE.2017.2663435. ↩↩

  4. Fraser, Gordon, and Andrea Arcuri. 2011. "EvoSuite: Automatic Test Suite Generation for Object-Oriented Software." Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE '11), 416–19. https://doi.org/10.1145/2025113.2025179. ↩

  5. Lukasczyk, Stephan, and Gordon Fraser. 2022. "Pynguin: Automated Unit Test Generation for Python." Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion '22), 168–72. https://doi.org/10.1145/3510454.3516829. ↩

  6. Lemieux, Caroline, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. "CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models." 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 919–31. https://doi.org/10.1109/ICSE48619.2023.00085. ↩

  7. Shamshiri, Sina, René Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. "Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges." 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 201–11. https://doi.org/10.1109/ASE.2015.86. ↩

  8. Fraser, Gordon, Matt Staats, Phil McMinn, Andrea Arcuri, and Frank Padberg. 2015. "Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study." ACM Transactions on Software Engineering and Methodology 24 (4): 23:1–49. https://doi.org/10.1145/2699688. ↩