# Testing machine-learning systems

Testing a machine-learning system means testing without an oracle: the model's
logic is learned from data, so for most inputs nobody knows the correct output.
The *training* procedure is authored and can be reviewed like any program; the
model it produces cannot be. That missing oracle is the central problem, and it
decides which methods apply.

This is ML *as the system under test*: the model is the artifact being
verified. It is the opposite direction from using a model to *do* the
testing — the question of AI as the oracle, taken up in the [AI
angle](https://quality.stereobooster.com/ai.md).

## What it catches

With no oracle, the useful questions shift from "is this output correct"
to properties that can be checked without one:

- **Behavioral bugs a single accuracy number hides.** A sentiment model
  at 95% accuracy can still flip its answer when a name is swapped or a
  clause negated. Capability-level tests catch the systematic failure the
  aggregate metric averages away.
- **Robustness failures.** Inputs a human would classify identically that
  the model does not: a rotated sign, an imperceptibly perturbed image, a
  paraphrase. These are reliability and security defects both.
- **Data and distribution defects.** A feature that silently went null, a
  serving distribution that drifted from the training distribution. Most
  production ML failures are data failures, not model-logic failures
  (Breck et al. 2019)[^breck2019].

What it does *not* catch is ordinary correctness against a specification,
because there is none. "Correct" is defined only statistically, over a
distribution, so no test certifies a specific output the way `assert
f(2) == 4` does for hand-written code.

## Pseudo-oracles: metamorphic and differential testing

[Metamorphic testing](https://quality.stereobooster.com/metamorphic-testing.md) checks how the
output should *change* under a transformation of the input, which needs
no reference answer. DeepTest generated tests for autonomous-driving
models this way: under brightness, rotation, fog, and rain
transformations that should not change the steering decision, it found
many inputs where the decision did change (Tian et al. 2018)[^tian2018]. Sentiment and
translation models get the same treatment with paraphrase and
negation relations.

[Differential testing](https://quality.stereobooster.com/differential-testing.md) treats a
disagreement between independently trained models for the same task as a
bug-revealing input, again with no reference output required. DeepXplore
combined this with a coverage-guided search for inputs that make models
disagree (Pei et al. 2017)[^pei2017]. Both are general-purpose methods pointed at a model
whose oracle is missing.

## Behavioral testing: capabilities, not one accuracy number

A held-out accuracy score is a single aggregate; it says nothing about
*which* capabilities the model has. CheckList reframes model evaluation
as capability-level test suites (Ribeiro et al. 2020)[^ribeiro2020]: for each capability
(negation, named entities, coreference), it writes *minimum-functionality*
tests (simple cases the model must pass), *invariance* tests (the output
should not change under a label-preserving edit, a metamorphic relation),
and *directional-expectation* tests (the output should move a known way).
Applying it to commercial NLP systems surfaced failures their headline
accuracy had hidden.

## Adversarial robustness: a property, not an example

Robustness is a property, not an expected value: a small perturbation of
the input should not change the prediction. Testing it means *searching*
for a counterexample, a nearby input that flips the output. The standard
attacks, from the single-step FGSM (Goodfellow et al. 2015)[^goodfellow2015] to iterated
projected gradient descent (Madry et al. 2018)[^madry2018], are gradient-guided searches
over the input's neighborhood, and libraries like [foolbox](https://github.com/bethgelab/foolbox)
and the [Adversarial Robustness Toolbox](https://github.com/Trusted-AI/adversarial-robustness-toolbox) package them for use
as a test. This is [search-based generation](https://quality.stereobooster.com/search-based-software-testing.md)
with a robustness [property](https://quality.stereobooster.com/property-based-testing.md) as the
oracle; a found perturbation is a concrete failing test.

## Formal verification of neural networks

[Formal methods](https://quality.stereobooster.com/formal.md) can *prove* robustness, not just
test it, within a bounded input region. Two families mirror the ones for
ordinary code:

- **SMT and search over the network.** Reluplex extended a linear-arithmetic
  SMT solver to reason about ReLU activations and prove (or refute, with a
  counterexample) that no input in a region changes the output
  (Katz et al. 2017)[^katz2017]; its successor Marabou is the maintained tool.
- **[Abstract interpretation](https://quality.stereobooster.com/abstract-interpretation.md).** AI²
  propagates an over-approximation of an input region through the network
  and checks the output stays in the safe set (Gehr et al. 2018)[^gehr2018]; the ERAN and
  α,β-CROWN tools are the modern incarnations.

A complementary line makes robustness *certifiable* by construction:
randomized smoothing turns any classifier into one with a provable
robustness radius around each input (Cohen et al. 2019)[^cohen2019]. The limit on all of
this is scale: sound verification is tractable for modest networks and
*local* properties (robustness in an ε-ball), not for "the model is
correct" over its whole domain.

## The data is the code

Because the model is fit to data, the data is where much of the real
behavior, and many of the bugs, live:

- **Schema and distribution checks** on training and serving data catch
  the silently-null feature and the drifted category before they reach the
  model. [TensorFlow Data Validation](https://github.com/tensorflow/data-validation) and
  [Great Expectations](https://github.com/great-expectations/great_expectations) express these as
  assertions over a dataset, the data analogue of unit tests
  (Breck et al. 2019)[^breck2019].
- **Distribution-shift monitoring** in production is the ML case of
  [monitoring](https://quality.stereobooster.com/monitoring-and-observability.md) and
  [change-point detection](https://quality.stereobooster.com/change-point-detection.md): the
  input distribution, not just the error rate, is the signal to watch,
  because a model degrades silently when the world moves away from its
  training data.

## Neuron coverage is not test adequacy

The most-cited ML-testing idea carries the field's clearest
folklore. DeepXplore introduced **neuron coverage**, the fraction of a
network's neurons activated by a test set, as a DNN analogue of code
coverage (Pei et al. 2017)[^pei2017], and DeepGauge extended it to a family of finer
criteria (Ma et al. 2018)[^ma2018]. The intuition, that more neurons exercised means a
more thorough test, is the same one behind code-coverage targets.

The evidence does not support it. Harel-Canada and colleagues tested
whether neuron coverage tracks what a test suite is supposed to do and
found it does not: higher neuron coverage was not associated with more
defects or adversarial examples found, and optimizing for it *reduced*
the diversity and naturalness of the generated inputs (Harel-Canada et al. 2020)[^harelcanada2020].
This is the [coverage-is-not-effectiveness](https://quality.stereobooster.com/measuring-test-effectiveness.md)
result again, in a new medium: neuron coverage measures a reachability-like
surface property, not the fault-detecting effectiveness of the suite, so chasing
the number optimizes the proxy instead of the goal. It is, at best, a
weak gap-finder, not an adequacy target.

## When to use, when not

**Use:**

- **Metamorphic and differential testing as the default**, because they
  are the techniques that work without an oracle and cover the most
  behavior for the least ceremony.
- **CheckList-style capability suites** for any NLP or classification
  model going to production, in place of trusting a single accuracy
  number.
- **Adversarial search** where an attacker is in the threat model, or
  where robustness is a stated requirement (safety-critical perception).
- **Data validation and drift monitoring always.** It is cheap and catches
  the most common real failures.
- **Formal NN verification** for small, safety-critical models where a
  *proof* of local robustness is worth its cost (aircraft collision
  avoidance is the canonical case).

**Don't:**

- **Chase a neuron-coverage target** as a measure of test quality; it does
  not track fault-finding.
- **Read held-out accuracy as a correctness test.** It is an aggregate
  over a distribution, not a verdict on any behavior, and it hides
  systematic capability gaps.
- **Expect formal verification to certify overall correctness.** It proves
  bounded, local properties, not that the model does the right thing
  everywhere.

## Tools

| Tool | Role | Notes |
| ---- | ---- | ----- |
| [foolbox](https://github.com/bethgelab/foolbox), [ART](https://github.com/Trusted-AI/adversarial-robustness-toolbox) | Adversarial testing | Libraries of attacks (FGSM, PGD, and many more) used to search for robustness counterexamples. |
| [Marabou](https://github.com/NeuralNetworkVerification/Marabou) | Formal verification | SMT-based verifier for ReLU networks; the maintained successor to Reluplex. |
| [ERAN](https://github.com/eth-sri/eran), [α,β-CROWN](https://github.com/Verified-Intelligence/alpha-beta-CROWN) | Formal verification | Abstract-interpretation and bound-propagation robustness verifiers; α,β-CROWN wins the recurring VNN-COMP verification competition. |
| [CheckList](https://github.com/marcotcr/checklist) | Behavioral testing | Capability-based test-suite framework for NLP models. |
| [TensorFlow Data Validation](https://github.com/tensorflow/data-validation), [Great Expectations](https://github.com/great-expectations/great_expectations) | Data validation | Schema, distribution, and expectation checks over training and serving data. |

## Evidence

- **Metamorphic and differential testing find real defects** in deployed
  model classes: driving models under image transformations (Tian et al. 2018)[^tian2018]
  and disagreements across models (Pei et al. 2017)[^pei2017].
- **Behavioral testing surfaces failures accuracy hides** (Ribeiro et al. 2020)[^ribeiro2020].
- **Neuron coverage does not measure test effectiveness** (Harel-Canada et al. 2020)[^harelcanada2020].
- **Scope.** These methods reduce the uncertainty a missing oracle leaves,
  they do not remove it. Robustness proofs are local and small-scale;
  metamorphic relations check only the invariances that were stated; data
  validation checks only the properties that were specified. None
  certifies that a learned model is correct.

## Further reading

- A survey of ML testing across properties, components, and workflow stages
  (Zhang et al. 2022)[^zhang2022] maps the primary literature behind these techniques.

## Classification

- **Quality dimensions:** Functionality (behavioral correctness of a model that has no reference oracle), Reliability (robustness to input perturbation and to distribution shift in production), Security (adversarial robustness: evasion by crafted perturbations) — a class of system under test, not a single method: the oracle problem routes it to metamorphic, differential, search-based, and formal methods, which carry the axes. Listed here for the dimensions ML testing serves, not for a coordinate of its own.
- **Area:** Machine-learning and deep-learning systems as the system under test: image and text classifiers, autonomous-driving perception, NLP models, recommenders and rankers — anywhere the program is learned from data and has no hand-written specification to check against.

## Referenced by

- [Metamorphic testing](https://quality.stereobooster.com/metamorphic-testing.md) · Methods
- [Search-based software testing (SBST)](https://quality.stereobooster.com/search-based-software-testing.md) · Methods
- [Testing autonomous and cyber-physical systems](https://quality.stereobooster.com/testing-autonomous-systems.md) · Methods

## References

[^breck2019]: Breck, Eric, Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. 2019. "[Data Validation for Machine Learning](https://proceedings.mlsys.org/paper_files/paper/2019/file/928f1160e52192e3e0017fb63ab65391-Paper.pdf)." *Proceedings of Machine Learning and Systems (MLSys 2019)* 1: 334–47. [https://proceedings.mlsys.org/paper\\\_files/paper/2019/file/928f1160e52192e3e0017fb63ab65391-Paper.pdf](https://proceedings.mlsys.org/paper\_files/paper/2019/file/928f1160e52192e3e0017fb63ab65391-Paper.pdf).
[^tian2018]: Tian, Yuchi, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. "[DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars](https://arxiv.org/pdf/1708.08559)." *Proceedings of the 40th International Conference on Software Engineering (ICSE '18)*, 303–14. <https://doi.org/10.1145/3180155.3180220>.
[^pei2017]: Pei, Kexin, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. "[DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://arxiv.org/pdf/1705.06640)." *Proceedings of the 26th Symposium on Operating Systems Principles (SOSP '17)*, 1–18. <https://doi.org/10.1145/3132747.3132785>.
[^ribeiro2020]: Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. "[Beyond Accuracy: Behavioral Testing of NLP Models with CheckList](https://arxiv.org/pdf/2005.04118)." *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)*, 4902–12. <https://doi.org/10.18653/v1/2020.acl-main.442>.
[^goodfellow2015]: Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. 2015. "[Explaining and Harnessing Adversarial Examples](https://ar5iv.labs.arxiv.org/html/1412.6572)." *3rd International Conference on Learning Representations (ICLR 2015)*. <https://doi.org/10.48550/arXiv.1412.6572>.
[^madry2018]: Madry, Aleksander, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. "[Towards Deep Learning Models Resistant to Adversarial Attacks](https://arxiv.org/pdf/1706.06083)." *6th International Conference on Learning Representations (ICLR 2018)*. <https://doi.org/10.48550/arXiv.1706.06083>.
[^katz2017]: Katz, Guy, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2017. "[Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://arxiv.org/pdf/1702.01135)." *Computer Aided Verification (CAV 2017)*, 97–117. [https://doi.org/10.1007/978-3-319-63387-9\\\_5](https://doi.org/10.1007/978-3-319-63387-9\_5).
[^gehr2018]: Gehr, Timon, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. 2018. "[AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation](https://ieeexplore.ieee.org/ielx7/8418581/8418583/08418593.pdf)." *2018 IEEE Symposium on Security and Privacy (SP)*, 3–18. <https://doi.org/10.1109/SP.2018.00058>.
[^cohen2019]: Cohen, Jeremy M., Elan Rosenfeld, and J. Zico Kolter. 2019. "[Certified Adversarial Robustness via Randomized Smoothing](https://arxiv.org/pdf/1902.02918)." *Proceedings of the 36th International Conference on Machine Learning (ICML 2019)*. <https://doi.org/10.48550/arXiv.1902.02918>.
[^ma2018]: Ma, Lei, Felix Juefei-Xu, Fuyuan Zhang, et al. 2018. "[DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://arxiv.org/pdf/1803.07519)." *Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018)*, 120–31. <https://doi.org/10.1145/3238147.3238202>.
[^harelcanada2020]: Harel-Canada, Fabrice, Lingxiao Wang, Muhammad Ali Gulzar, Quanquan Gu, and Miryung Kim. 2020. "[Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks?](https://dl.acm.org/doi/pdf/10.1145/3368089.3409754)" *Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020)*, 851–62. <https://doi.org/10.1145/3368089.3409754>.
[^zhang2022]: Zhang, Jie M., Mark Harman, Lei Ma, and Yang Liu. 2022. "[Machine Learning Testing: Survey, Landscapes and Horizons](https://arxiv.org/pdf/1906.10742)." *IEEE Transactions on Software Engineering* 48 (1): 1–36. <https://doi.org/10.1109/TSE.2019.2962027>.

## Acronyms

- DNN — deep neural network
- FGSM — fast gradient sign method
- NLP — natural language processing
- NN — neural network
- PGD — projected gradient descent
- SBST — search-based software testing
- SMT — satisfiability modulo theories
- VNN-COMP — Verification of Neural Networks Competition