Skip to content

Software Quality

Testing machine-learning systems

Testing a machine-learning system means testing without an oracle: the model's logic is learned from data, so for most inputs nobody knows the correct output. The training procedure is authored and can be reviewed like any program; the model it produces cannot be. That missing oracle is the central problem, and it decides which methods apply.

This is ML as the system under test: the model is the artifact being verified. It is the opposite direction from using a model to do the testing — the question of AI as the oracle, taken up in the AI angle.

What it catches

With no oracle, the useful questions shift from "is this output correct" to properties that can be checked without one:

  • Behavioral bugs a single accuracy number hides. A sentiment model at 95% accuracy can still flip its answer when a name is swapped or a clause negated. Capability-level tests catch the systematic failure the aggregate metric averages away.
  • Robustness failures. Inputs a human would classify identically that the model does not: a rotated sign, an imperceptibly perturbed image, a paraphrase. These are reliability and security defects both.
  • Data and distribution defects. A feature that silently went null, a serving distribution that drifted from the training distribution. Most production ML failures are data failures, not model-logic failures (Breck et al. 2019)1.

What it does not catch is ordinary correctness against a specification, because there is none. "Correct" is defined only statistically, over a distribution, so no test certifies a specific output the way assert f(2) == 4 does for hand-written code.

Pseudo-oracles: metamorphic and differential testing

Metamorphic testing checks how the output should change under a transformation of the input, which needs no reference answer. DeepTest generated tests for autonomous-driving models this way: under brightness, rotation, fog, and rain transformations that should not change the steering decision, it found many inputs where the decision did change (Tian et al. 2018)2. Sentiment and translation models get the same treatment with paraphrase and negation relations.

Differential testing treats a disagreement between independently trained models for the same task as a bug-revealing input, again with no reference output required. DeepXplore combined this with a coverage-guided search for inputs that make models disagree (Pei et al. 2017)3. Both are general-purpose methods pointed at a model whose oracle is missing.

Behavioral testing: capabilities, not one accuracy number

A held-out accuracy score is a single aggregate; it says nothing about which capabilities the model has. CheckList reframes model evaluation as capability-level test suites (Ribeiro et al. 2020)4: for each capability (negation, named entities, coreference), it writes minimum-functionality tests (simple cases the model must pass), invariance tests (the output should not change under a label-preserving edit, a metamorphic relation), and directional-expectation tests (the output should move a known way). Applying it to commercial NLP systems surfaced failures their headline accuracy had hidden.

Adversarial robustness: a property, not an example

Robustness is a property, not an expected value: a small perturbation of the input should not change the prediction. Testing it means searching for a counterexample, a nearby input that flips the output. The standard attacks, from the single-step FGSM (Goodfellow et al. 2015)5 to iterated projected gradient descent (Madry et al. 2018)6, are gradient-guided searches over the input's neighborhood, and libraries like foolbox and the Adversarial Robustness Toolbox package them for use as a test. This is search-based generation with a robustness property as the oracle; a found perturbation is a concrete failing test.

Formal verification of neural networks

Formal methods can prove robustness, not just test it, within a bounded input region. Two families mirror the ones for ordinary code:

  • SMT and search over the network. Reluplex extended a linear-arithmetic SMT solver to reason about ReLU activations and prove (or refute, with a counterexample) that no input in a region changes the output (Katz et al. 2017)7; its successor Marabou is the maintained tool.
  • Abstract interpretation. AI² propagates an over-approximation of an input region through the network and checks the output stays in the safe set (Gehr et al. 2018)8; the ERAN and α,β-CROWN tools are the modern incarnations.

A complementary line makes robustness certifiable by construction: randomized smoothing turns any classifier into one with a provable robustness radius around each input (Cohen et al. 2019)9. The limit on all of this is scale: sound verification is tractable for modest networks and local properties (robustness in an ε-ball), not for "the model is correct" over its whole domain.

The data is the code

Because the model is fit to data, the data is where much of the real behavior, and many of the bugs, live:

  • Schema and distribution checks on training and serving data catch the silently-null feature and the drifted category before they reach the model. TensorFlow Data Validation and Great Expectations express these as assertions over a dataset, the data analogue of unit tests (Breck et al. 2019)1.
  • Distribution-shift monitoring in production is the ML case of monitoring and change-point detection: the input distribution, not just the error rate, is the signal to watch, because a model degrades silently when the world moves away from its training data.

Neuron coverage is not test adequacy

The most-cited ML-testing idea carries the field's clearest folklore. DeepXplore introduced neuron coverage, the fraction of a network's neurons activated by a test set, as a DNN analogue of code coverage (Pei et al. 2017)3, and DeepGauge extended it to a family of finer criteria (Ma et al. 2018)10. The intuition, that more neurons exercised means a more thorough test, is the same one behind code-coverage targets.

The evidence does not support it. Harel-Canada and colleagues tested whether neuron coverage tracks what a test suite is supposed to do and found it does not: higher neuron coverage was not associated with more defects or adversarial examples found, and optimizing for it reduced the diversity and naturalness of the generated inputs (Harel-Canada et al. 2020)11. This is the coverage-is-not-effectiveness result again, in a new medium: neuron coverage measures a reachability-like surface property, not the fault-detecting effectiveness of the suite, so chasing the number optimizes the proxy instead of the goal. It is, at best, a weak gap-finder, not an adequacy target.

When to use, when not

Use:

  • Metamorphic and differential testing as the default, because they are the techniques that work without an oracle and cover the most behavior for the least ceremony.
  • CheckList-style capability suites for any NLP or classification model going to production, in place of trusting a single accuracy number.
  • Adversarial search where an attacker is in the threat model, or where robustness is a stated requirement (safety-critical perception).
  • Data validation and drift monitoring always. It is cheap and catches the most common real failures.
  • Formal NN verification for small, safety-critical models where a proof of local robustness is worth its cost (aircraft collision avoidance is the canonical case).

Don't:

  • Chase a neuron-coverage target as a measure of test quality; it does not track fault-finding.
  • Read held-out accuracy as a correctness test. It is an aggregate over a distribution, not a verdict on any behavior, and it hides systematic capability gaps.
  • Expect formal verification to certify overall correctness. It proves bounded, local properties, not that the model does the right thing everywhere.

Tools

Tool Role Notes
foolbox, ART Adversarial testing Libraries of attacks (FGSM, PGD, and many more) used to search for robustness counterexamples.
Marabou Formal verification SMT-based verifier for ReLU networks; the maintained successor to Reluplex.
ERAN, α,β-CROWN Formal verification Abstract-interpretation and bound-propagation robustness verifiers; α,β-CROWN wins the recurring VNN-COMP verification competition.
CheckList Behavioral testing Capability-based test-suite framework for NLP models.
TensorFlow Data Validation, Great Expectations Data validation Schema, distribution, and expectation checks over training and serving data.

Evidence

  • Metamorphic and differential testing find real defects in deployed model classes: driving models under image transformations (Tian et al. 2018)2 and disagreements across models (Pei et al. 2017)3.
  • Behavioral testing surfaces failures accuracy hides (Ribeiro et al. 2020)4.
  • Neuron coverage does not measure test effectiveness (Harel-Canada et al. 2020)11.
  • Scope. These methods reduce the uncertainty a missing oracle leaves, they do not remove it. Robustness proofs are local and small-scale; metamorphic relations check only the invariances that were stated; data validation checks only the properties that were specified. None certifies that a learned model is correct.

Further reading

  • A survey of ML testing across properties, components, and workflow stages (Zhang et al. 2022)12 maps the primary literature behind these techniques.

Classification

  • Quality dimensions: Functionality (behavioral correctness of a model that has no reference oracle), Reliability (robustness to input perturbation and to distribution shift in production), Security (adversarial robustness: evasion by crafted perturbations) — a class of system under test, not a single method: the oracle problem routes it to metamorphic, differential, search-based, and formal methods, which carry the axes. Listed here for the dimensions ML testing serves, not for a coordinate of its own.
  • Area: Machine-learning and deep-learning systems as the system under test: image and text classifiers, autonomous-driving perception, NLP models, recommenders and rankers — anywhere the program is learned from data and has no hand-written specification to check against.

Referenced by

References

  1. Breck, Eric, Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. 2019. "Data Validation for Machine Learning." Proceedings of Machine Learning and Systems (MLSys 2019) 1: 334–47. https://proceedings.mlsys.org/paper_files/paper/2019/file/928f1160e52192e3e0017fb63ab65391-Paper.pdf. ↩↩

  2. Tian, Yuchi, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. "DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars." Proceedings of the 40th International Conference on Software Engineering (ICSE '18), 303–14. https://doi.org/10.1145/3180155.3180220. ↩↩

  3. Pei, Kexin, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. "DeepXplore: Automated Whitebox Testing of Deep Learning Systems." Proceedings of the 26th Symposium on Operating Systems Principles (SOSP '17), 1–18. https://doi.org/10.1145/3132747.3132785. ↩↩↩

  4. Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), 4902–12. https://doi.org/10.18653/v1/2020.acl-main.442. ↩↩

  5. Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. 2015. "Explaining and Harnessing Adversarial Examples." 3rd International Conference on Learning Representations (ICLR 2015). https://doi.org/10.48550/arXiv.1412.6572. ↩

  6. Madry, Aleksander, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. "Towards Deep Learning Models Resistant to Adversarial Attacks." 6th International Conference on Learning Representations (ICLR 2018). https://doi.org/10.48550/arXiv.1706.06083. ↩

  7. Katz, Guy, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2017. "Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks." Computer Aided Verification (CAV 2017), 97–117. https://doi.org/10.1007/978-3-319-63387-9_5. ↩

  8. Gehr, Timon, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. 2018. "AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation." 2018 IEEE Symposium on Security and Privacy (SP), 3–18. https://doi.org/10.1109/SP.2018.00058. ↩

  9. Cohen, Jeremy M., Elan Rosenfeld, and J. Zico Kolter. 2019. "Certified Adversarial Robustness via Randomized Smoothing." Proceedings of the 36th International Conference on Machine Learning (ICML 2019). https://doi.org/10.48550/arXiv.1902.02918. ↩

  10. Ma, Lei, Felix Juefei-Xu, Fuyuan Zhang, et al. 2018. "DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems." Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018), 120–31. https://doi.org/10.1145/3238147.3238202. ↩

  11. Harel-Canada, Fabrice, Lingxiao Wang, Muhammad Ali Gulzar, Quanquan Gu, and Miryung Kim. 2020. "Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks?" Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020), 851–62. https://doi.org/10.1145/3368089.3409754. ↩↩

  12. Zhang, Jie M., Mark Harman, Lei Ma, and Yang Liu. 2022. "Machine Learning Testing: Survey, Landscapes and Horizons." IEEE Transactions on Software Engineering 48 (1): 1–36. https://doi.org/10.1109/TSE.2019.2962027. ↩