Skip to content

Software Quality

Testing GUI and mobile applications

Testing a GUI application means driving it through its interface, because that is the only way in. The app is a stateful graph of screens joined by events: taps, swipes, text, rotation, the back button, lifecycle pauses and resumes. Two ways to drive it exist. A person can write scripts that drive the interface, which is nothing new: a Selenium, Espresso, or Playwright test is an example test whose input is an event sequence and whose oracle reads the rendered UI. The distinctive move is the other one: let a tool drive the app by itself and watch for it to crash or hang.

What it catches

  • Crashes and ANRs under event sequences no one scripted. Automated exploration fires long, unusual streams of events and finds the app states that crash or hang, the failures a scripted suite never thought to reach.
  • Visual regressions. Screenshot comparison catches them — a visual snapshot taken in-process by Playwright and most UI frameworks (toHaveScreenshot).

What it does not catch is functional wrongness. The oracle of automated exploration is the crash and the ANR; it establishes that the app ran without dying, never that it did the right thing. Correctness through the UI is the job of the scripted example tests, with their hand-written assertions.

Automated GUI exploration

The technique that makes this a domain of its own is generating the event sequences automatically. It is fuzzing pointed at a UI: random or guided input, an implicit crash oracle. Three strategies, in rising sophistication:

  • Random ("monkey") testing. Android's built-in Monkey fires a pseudo-random stream of events at the app.
  • Model-based exploration. Infer a model of the GUI (a state and event-flow graph) and traverse it systematically; Stoat drives this with a stochastic model it refines as it explores (Su et al. 2017)1.
  • Search-based exploration. Sapienz casts it as a search-based multi-objective problem: maximize coverage and crashes while minimizing the length of the event sequence, so the crash reports it files are short enough to act on (Mao et al. 2016)2. Facebook deployed Sapienz on its apps at scale (Alshahwan et al. 2018)3.

Random is the baseline to beat

The guided strategies do not beat the random one. Choudhary, Gorla, and Orso compared the Android test-generation tools head to head and found plain Monkey was as effective as or better than the model-based and search-based tools on both code coverage and crash detection, while being faster and needing no model (Choudhary et al. 2015)4.

The strength and the ceiling share a cause. Random exploration is fast, unbiased, and needs no model, so it cheaply reaches most states that a few taps can reach. But coverage plateaus well short of the whole app, because random events cannot pass an input gate: they will not type a valid password, complete a checkout, or work through a multi-step wizard, so everything behind meaningful input stays invisible. And the crash-only oracle means even full exploration would only ever find crashes, not wrong behavior. Automated exploration is a cheap, effective robustness gate.

On the web

The technique ports to the browser, but without a maintained equivalent of Monkey. gremlins.js is the closest analogue: it runs random agents against the page, watching for thrown exceptions and frame-rate collapse as its crash oracle. Crawljax sits at the guided end, firing events to build a state-flow graph, closer to model-based exploration than to random. Both projects have stalled, so a web monkey today is usually a random driver layered on a maintained browser framework, with gremlins.js as an optional event generator that still runs. The ceiling is the same one random exploration hits on mobile: it finds crashes on the surface reachable without input, not behavior behind a login or a multi-step form.

Device fragmentation and flakiness

  • Device fragmentation is conceptually ordinary: an app must work across a large matrix of device, OS-version, and screen-size combinations, which is a combinatorial covering-array problem run on device farms.
  • Flakiness is the practical tax on UI testing, and its main source is asynchrony rather than anything visual: improper waits, test-order dependency, and concurrency lead the measured causes (Luo et al. 2014)5. That analysis is not UI-specific, so it names the mechanism without sizing how much worse UI tests are than others. Their slowness has mechanical causes: driving a browser, a network round trip, and a render inside every action. Both together are the practical half of the pyramid critique: a suite top-heavy with UI tests is slow and noisy. It bears on regression-suite management too, since flaky tests corrupt the failure history that prioritization and predictive selection depend on.

Tools

Tool Role Notes
Monkey Random exploration Android's built-in event fuzzer; the baseline crash/robustness gate.
Appium Scripted, cross-platform WebDriver-based; drives Android and iOS from one script API.
Espresso / XCUITest Scripted, native In-process UI test frameworks for Android and iOS respectively.
Selenium / Playwright Scripted, web WebDriver (the W3C standard) and its modern successor for browser UIs.
gremlins.js Random exploration, web In-browser monkey (click, scroll, type, forms); dependency-free but unreleased since 2020, typically run inside Playwright or Cypress.
Crawljax Guided crawling, web Event-driven state exploration for JavaScript apps; releases stalled since 2023.

Device farms (Firebase Test Lab, AWS Device Farm, BrowserStack) are commercial SaaS that host the physical-device matrix.

When to use, when not

Use:

  • Automated exploration (start with Monkey) as a cheap crash and ANR gate before release; it finds robustness bugs no scripted suite reaches.
  • Scripted example tests for the functional flows that matter, since exploration cannot check correctness or pass input gates.
  • A combinatorial device matrix on a device farm where fragmentation is a real risk.
  • Visual snapshots for UI-regression-sensitive surfaces.

Don't:

  • Expect exploration to verify behavior. Its oracle is "did not crash"; correctness needs an oracle a person wrote.
  • Build a suite top-heavy with slow, flaky UI tests. The pyramid critique is about exactly this failure mode.
  • Read a coverage plateau as thoroughness. Automated exploration stalls at the first input gate, leaving the app's logic-heavy interior untouched.

Evidence

  • Random exploration is a strong baseline: as effective as the model-based and search-based Android tools on coverage and crash detection in a head-to-head comparison, and cheaper (Choudhary et al. 2015)4.
  • Search-based exploration works at industrial scale: Sapienz files short, actionable crash reports and ran on Facebook's apps (Mao et al. 2016; Alshahwan et al. 2018)2 3.
  • Scope. Automated GUI testing reduces the uncertainty that an app will crash or hang under real use; it says little about functional correctness, which still needs scripted oracles, and it cannot reach the states behind meaningful input. A systematic mapping of the field (Tramontana et al. 2019)6 surveys the fuller landscape.

Classification

  • Quality dimensions: Reliability (crash and ANR robustness under unscripted event sequences), Functionality (functional correctness, via scripted example tests through the UI and visual snapshots), Performance (responsiveness; an ANR is where reliability meets performance) — a class of system under test whose one distinctive technique is automated GUI exploration (the axes below describe it). Scripted UI tests are example tests reached through a UI, visual checks are approval tests, and the device matrix is a combinatorial problem: details that route to existing methods rather than new ones.
  • Area: Interactive GUI applications as the system under test, reached only through their interface: Android and iOS mobile apps (the research-rich core) and desktop and web front-ends. Event-driven, stateful, and fragmented across devices.
  • Guarantee: Empirical — a search over a vast event space; a crash found is real, but coverage plateaus well short of the whole app and finding none proves nothing.

Referenced by

References

  1. Su, Ting, Guozhu Meng, Yuting Chen, et al. 2017. "Guided, Stochastic Model-Based GUI Testing of Android Apps." Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017), 245–56. https://doi.org/10.1145/3106237.3106298. ↩

  2. Mao, Ke, Mark Harman, and Yue Jia. 2016. "Sapienz: Multi-objective Automated Testing for Android Applications." Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016), 94–105. https://doi.org/10.1145/2931037.2931054. ↩↩

  3. Alshahwan, Nadia, Xinbo Gao, Mark Harman, et al. 2018. "Deploying Search Based Software Engineering with Sapienz at Facebook." Search-Based Software Engineering (SSBSE 2018), 3–45. https://doi.org/10.1007/978-3-319-99241-9_1. ↩↩

  4. Choudhary, Shauvik Roy, Alessandra Gorla, and Alessandro Orso. 2015. "Automated Test Input Generation for Android: Are We There Yet?" 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 429–40. https://doi.org/10.48550/arXiv.1503.07217. ↩↩

  5. Luo, Qingzhou, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. "An Empirical Analysis of Flaky Tests." Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '14) (Hong Kong, China), 643–53. https://doi.org/10.1145/2635868.2635920. ↩

  6. Tramontana, Porfirio, Domenico Amalfitano, Nicola Amatucci, and Anna Rita Fasolino. 2019. "Automated Functional Testing of Mobile Applications: A Systematic Mapping Study." Software Quality Journal 27 (1): 149–201. https://doi.org/10.1007/s11219-018-9418-6. ↩