Skip to content

Software Quality

Testing autonomous and cyber-physical systems

Testing an autonomous or cyber-physical system means showing it stays safe across the situations it will meet in the open world. Self-driving cars, drones, mobile robots and the control software that drives them all face the same difficulty, and it is scale: that space of situations is combinatorially vast, and the events that matter, crashes, are extraordinarily rare. A RAND analysis showed that proving a self-driving car statistically safer than a human at the fatality rate would take hundreds of millions to billions of miles of driving, and again after every software change (Kalra and Paddock 2016)1. On-road mileage alone cannot get there.

Learned perception is usually part of the stack, so ML's missing-oracle problem (no known-correct output to test against) is present too.

What it catches

  • Failures in situations no one drove into by hand. A structured scenario space reaches the unprotected left turn into low sun, the pedestrian stepping from behind a truck, the cut-in at a closing speed, cases too rare to encounter in a fleet's real miles but constructible in simulation.
  • Safety-envelope violations. Not just crashes but the precursors: following too close, a time-to-collision below threshold, a maneuver outside a formally safe region.
  • The rare critical event, found faster than reality offers it. Search and importance-sampling concentrate effort on the small fraction of scenarios that stress the system.

What it does not catch is the scenario no one modeled. Scenario-based testing covers the scenarios and parameter values that were enumerated; the residue, the "unknown-unsafe" situations, is the central open problem the safety standards name explicitly.

Scenario-based testing: structuring the space

Instead of random miles, the field tests over a space of scenarios described at three levels of abstraction, the vocabulary from Germany's PEGASUS project: functional (a natural-language situation), logical (parameter ranges over it), and concrete (specific sampled values). ASAM OpenSCENARIO is the vendor-neutral interchange standard for the dynamic content (actors, maneuvers, triggers) over an OpenDRIVE road network.

Sampling a logical scenario's parameter space is a combinatorial-testing problem: the axes (speed, gap, weather, lighting) interact, and a covering array over them tests interactions without the full product. Higher up, Scenic is a probabilistic scenario-specification language (Fremont et al. 2019)2: it defines a distribution over scenes and scenarios that a tool samples, and the same specification feeds the falsification search.

Simulation

Exercising these scenarios cheaply means simulation. CARLA (Dosovitskiy et al. 2017)3 is the dominant open simulator (sensor models, traffic, weather, an Unreal-based world); AirSim and BeamNG are the other open options, and industrial practice runs largely on commercial and in-house stacks: NVIDIA DRIVE Sim, Applied Intuition, and Waymo's internal simulation.

  • The sim-to-real gap. A defect found in simulation may not reproduce on the road, and passing in simulation is not evidence of real safety. Fidelity of sensors, dynamics, and other road users decides what transfers, and it is never complete.
  • This is not deterministic simulation testing. Deterministic simulation testing shares the word and pursues a different goal: it runs a distributed system under a deterministic scheduler to make concurrency bugs reproducible. Autonomous-system simulation seeks physical fidelity to explore a scenario space. A reproducible seed is still worth borrowing, to make a found scenario re-runnable.

Finding the dangerous scenarios

Uniform sampling spends almost its whole budget on safe, uneventful driving.

  • Search-based scenario generation is the autonomous-driving instance of search-based test generation. AsFault evolves road networks that break lane-keeping using genetic search over procedurally generated tracks (Gambi et al. 2019)4; multi-objective search finds scenarios that stress several failure modes at once (Ben Abdessalem et al. 2018)5.
  • Temporal-logic falsification treats a safety requirement as a signal-temporal-logic specification and searches for an input trajectory that violates it, guided by the requirement's robustness (Fainekos and Pappas 2009)6. S-TaLiRo and Breach are the tools, and Berkeley's VerifAI couples the search to Scenic scenarios.
  • Accelerated evaluation reshapes the sampling distribution toward rare hazards. Training an adversarial driving environment with dense reinforcement learning surfaced safety-critical events orders of magnitude faster than naturalistic driving (Feng et al. 2023)7, recovering an unbiased risk estimate by correcting for the reweighting.

The oracle: safety envelopes

A crash is an unambiguous but rare and late oracle. The field supplements it with formal safety envelopes that fail earlier and more often: time-to-collision thresholds, and Responsibility-Sensitive Safety (RSS), Mobileye's formal model of safe distances and right-of-way with proven properties (Shalev-Shwartz et al. 2017)8. An envelope like RSS serves twice: as a test oracle in simulation, and as a runtime monitor on the road, a formal predicate over the live trajectory.

Assurance: what "tested enough" means

Because no test suite proves an autonomous system safe, the field frames the claim as a safety argument, and the standards carry it:

  • ISO 21448 (SOTIF) (ISO 2022)9, Safety Of The Intended Functionality, addresses hazards from performance limitations with no component fault present: the sensor that simply cannot resolve this scene. Its partition into known-safe, known-unsafe, and unknown-unsafe scenarios is the formal name for the scenario-coverage gap.
  • UL 4600 is a safety-case standard (Underwriters Laboratories 2023)10: rather than prescribe tests, it demands a structured claims-argument-evidence case for why the product is acceptably safe, with the gaps made explicit.
  • ISO 26262 remains the substrate for faults in the electronics and software, the classical functional-safety layer beneath SOTIF.

Koopman and Wagner catalog why autonomous-vehicle validation resists ordinary testing: complex requirements, non-deterministic and learned components, and target failure rates so low that brute-force testing cannot reach them (Koopman and Wagner 2016)11. Field metrics offered as a shortcut mislead. The standing example is the disengagement report: counts of human takeovers, filed with the California DMV, are routinely read as a safety ranking, but the reports use inconsistent definitions across companies and are gameable (a firm that tests on easy roads, or redefines a disengagement, reports better numbers), so the counts are not comparable across programs (Favarò et al. 2018)12. Disengagement rate is folklore-tier evidence, not a safety measure.

When to use, when not

Use:

  • Scenario-based simulation as the backbone of any autonomous-driving or robotics test program: it is the only way to reach the rare situations at the volume safety requires.
  • Falsification and search-based generation to find critical scenarios, rather than sampling uniformly and hoping to hit them.
  • A formal safety envelope (RSS or equivalent) as both a simulation oracle and a runtime monitor.
  • Accelerated / importance sampling when an unbiased rate estimate is needed, not just example failures.

Don't:

  • Read disengagement rate, or raw miles driven, as a safety metric.
  • Treat a green simulation run as safety. The sim-to-real gap means it is necessary, not sufficient; pair it with closed-course and monitored on-road testing.
  • Mistake scenario coverage for completeness. A catalog covers only the scenarios enumerated; the unknown-unsafe residue needs a safety case.

Tools

Tool Role Notes
CARLA Simulation Open-source urban-driving simulator with sensor, traffic, and weather models; the research default.
ASAM OpenSCENARIO Scenario description Vendor-neutral standard for dynamic scenario content over an OpenDRIVE road network.
Scenic Scenario generation Probabilistic scenario-specification language; samples concrete scenes and feeds falsification.

Evidence

  • On-road miles alone cannot demonstrate safety at the required confidence (Kalra and Paddock 2016)1.
  • Search and falsification find critical scenarios that uniform sampling misses, across lane-keeping (Gambi et al. 2019)4 and multi-objective scenario stress (Ben Abdessalem et al. 2018)5, on the robustness-semantics foundation that makes the search tractable (Fainekos and Pappas 2009)6.
  • Sampling can be accelerated without bias, surfacing rare hazards far faster than naturalistic driving while preserving the risk estimate (Feng et al. 2023)7.
  • Field metrics can mislead: disengagement reports use inconsistent definitions across companies, so their counts are not comparable across programs (Favarò et al. 2018)12.
  • Scope. These methods reduce the uncertainty left by an untestable scenario space; they do not eliminate it. Simulation is limited by fidelity, scenario testing by the scenarios enumerated, falsification by the properties formalized. What remains, the unknown-unsafe region, is handled by a safety argument (SOTIF, UL 4600), not by a test that certifies the system safe everywhere.

Classification

  • Quality dimensions: Functionality: safety (hazard-free behavior across a scenario space, verified against a safety envelope), Reliability: safety (robustness to rare and adversarial operating conditions the fleet will eventually meet) — a class of system under test, not a single method: the scale-and-rare-event problem routes it to simulation, scenario generation, search-based falsification, and safety analysis, which carry the axes. Listed here for the dimensions it serves, not for a coordinate of its own.
  • Area: Autonomous and cyber-physical systems as the system under test: self-driving cars and driver-assistance, drones and mobile robots, and the control software whose safety turns on a vast space of physical operating scenarios.

Referenced by

References

  1. Kalra, Nidhi, and Susan M. Paddock. 2016. "Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?" Transportation Research Part A: Policy and Practice 94: 182–93. https://doi.org/10.1016/j.tra.2016.09.010. ↩↩

  2. Fremont, Daniel J., Tommaso Dreossi, Shromona Ghosh, Xiangyu Yue, Alberto L. Sangiovanni-Vincentelli, and Sanjit A. Seshia. 2019. "Scenic: A Language for Scenario Specification and Scene Generation." Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019), 63–78. https://doi.org/10.1145/3314221.3314633. ↩

  3. Dosovitskiy, Alexey, German Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun. 2017. "CARLA: An Open Urban Driving Simulator." Proceedings of the 1st Annual Conference on Robot Learning (CoRL 2017), 1–16. https://doi.org/10.48550/arXiv.1711.03938. ↩

  4. Gambi, Alessio, Marc Mueller, and Gordon Fraser. 2019. "Automatically Testing Self-Driving Cars with Search-Based Procedural Content Generation." Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019), 318–28. https://doi.org/10.1145/3293882.3330566. ↩↩

  5. Ben Abdessalem, Raja, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. "Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms." Proceedings of the 40th International Conference on Software Engineering (ICSE '18), 1016–26. https://doi.org/10.1145/3180155.3180160. ↩↩

  6. Fainekos, Georgios E., and George J. Pappas. 2009. "Robustness of Temporal Logic Specifications for Continuous-Time Signals." Theoretical Computer Science 410 (42): 4262–91. https://doi.org/10.1016/j.tcs.2009.06.021. ↩↩

  7. Feng, Shuo, Haowei Sun, Xintao Yan, et al. 2023. "Dense Reinforcement Learning for Safety Validation of Autonomous Vehicles." Nature 615: 620–27. https://doi.org/10.1038/s41586-023-05732-2. ↩↩

  8. Shalev-Shwartz, Shai, Shaked Shammah, and Amnon Shashua. 2017. On a Formal Model of Safe and Scalable Self-driving Cars. https://doi.org/10.48550/arXiv.1708.06374. ↩

  9. ISO. 2022. ISO 21448: Road Vehicles — Safety of the Intended Functionality (SOTIF). International Organization for Standardization. https://www.iso.org/standard/77490.html. ↩

  10. Underwriters Laboratories. 2023. ANSI/UL 4600: Standard for Safety for the Evaluation of Autonomous Products. Underwriters Laboratories. https://www.shopulstandards.com/ProductDetail.aspx?productid=UL4600. ↩

  11. Koopman, Philip, and Michael Wagner. 2016. "Challenges in Autonomous Vehicle Testing and Validation." SAE World Congress (SAE International Journal of Transportation Safety), 15–24. https://doi.org/10.4271/2016-01-0128. ↩

  12. Favarò, Francesca, Sky Eurich, and Nazanin Nader. 2018. "Autonomous Vehicles' Disengagements: Trends, Triggers, and Regulatory Limitations." Accident Analysis & Prevention 110: 136–48. https://doi.org/10.1016/j.aap.2017.11.001. ↩↩