# Testing GUI and mobile applications

Testing a GUI application means driving it through its interface, because
that is the only way in. The app is a stateful graph of screens joined by
events: taps, swipes, text, rotation, the back button, lifecycle pauses
and resumes. Two ways to drive it exist. A person can write scripts that
drive the interface, which is nothing new: a Selenium, Espresso, or
[Playwright](https://quality.stereobooster.com/playwright.md) test is an [example
test](https://quality.stereobooster.com/example-tests.md) whose input is an event sequence and
whose oracle reads the rendered UI. The distinctive move is the other one:
let a **tool drive the app by itself** and watch for it to crash or hang.

## What it catches

- **Crashes and ANRs under event sequences no one scripted.** Automated
  exploration fires long, unusual streams of events and finds the app
  states that crash or hang, the failures a scripted suite never thought
  to reach.
- **Visual regressions.** Screenshot comparison catches them — a [visual
  snapshot](https://quality.stereobooster.com/snapshot-testing.md#visual-snapshot) taken in-process by
  [Playwright](https://quality.stereobooster.com/playwright.md) and most UI frameworks (`toHaveScreenshot`).

What it does *not* catch is functional wrongness. The oracle of automated
exploration is the crash and the ANR; it establishes that the app *ran*
without dying, never that it did the *right* thing. Correctness through
the UI is the job of the scripted example tests, with their hand-written
assertions.

## Automated GUI exploration

The technique that makes this a domain of its own is generating the event
sequences automatically. It is [fuzzing](https://quality.stereobooster.com/fuzzing.md) pointed at a
UI: random or guided input, an implicit crash oracle. Three strategies,
in rising sophistication:

- **Random ("monkey") testing.** Android's built-in **Monkey** fires a
  pseudo-random stream of events at the app.
- **Model-based exploration.** Infer a model of the GUI (a state and
  event-flow graph) and traverse it systematically; Stoat drives this with
  a stochastic model it refines as it explores (Su et al. 2017)[^su2017].
- **Search-based exploration.** **Sapienz** casts it as a
  [search-based](https://quality.stereobooster.com/search-based-software-testing.md) multi-objective
  problem: maximize coverage and crashes while *minimizing* the length of
  the event sequence, so the crash reports it files are short enough to
  act on (Mao et al. 2016)[^mao2016]. Facebook deployed Sapienz on its apps at scale
  (Alshahwan et al. 2018)[^alshahwan2018].

## Random is the baseline to beat

The guided strategies do not beat the random one. Choudhary, Gorla,
and Orso compared the Android test-generation tools head to head and found
plain **Monkey was as effective as or better than** the model-based and
search-based tools on both code coverage and crash detection, while being
faster and needing no model (Choudhary et al. 2015)[^choudhary2015].

The strength and the ceiling share a cause. Random exploration is fast,
unbiased, and needs no model, so it cheaply reaches most states that a few
taps can reach. But coverage plateaus well short of the whole app, because
random events cannot pass an *input gate*: they will not type a valid
password, complete a checkout, or work through a multi-step wizard, so
everything behind meaningful input stays invisible. And the crash-only
oracle means even full exploration would only ever find crashes, not wrong
behavior. Automated exploration is a cheap, effective robustness gate.

## On the web

The technique ports to the browser, but without a maintained equivalent of
Monkey. [gremlins.js](https://github.com/marmelab/gremlins.js) is the closest analogue: it runs random
agents against the page, watching for thrown exceptions and frame-rate
collapse as its crash oracle. [Crawljax](https://github.com/crawljax/crawljax) sits at the guided
end, firing events to build a state-flow graph, closer to model-based
exploration than to random. Both projects have stalled, so a web monkey today
is usually a random driver layered on a maintained browser framework, with
gremlins.js as an optional event generator that still runs. The ceiling is the
same one random exploration hits on mobile: it finds crashes on the surface
reachable without input, not behavior behind a login or a multi-step form.

## Device fragmentation and flakiness

- **Device fragmentation** is conceptually ordinary: an app must work
  across a large matrix of device, OS-version, and screen-size
  combinations, which is a [combinatorial](https://quality.stereobooster.com/combinatorial-testing.md)
  covering-array problem run on device farms.
- **Flakiness** is the practical tax on UI testing, and its main source is
  asynchrony rather than anything visual: improper waits, test-order dependency,
  and concurrency lead the measured causes (Luo et al. 2014)[^luo2014]. That analysis is not
  UI-specific, so it names the mechanism without sizing how much worse UI tests
  are than others. Their slowness has mechanical causes: driving a browser, a
  network round trip, and a render inside every action. Both together are the
  practical half of the
  [pyramid critique](https://quality.stereobooster.com/testing-folklore.md): a suite
  top-heavy with UI tests is slow and noisy. It bears on
  [regression-suite management](https://quality.stereobooster.com/regression-management.md) too,
  since flaky tests corrupt the failure history that prioritization and
  predictive selection depend on.

## Tools

| Tool | Role | Notes |
| ---- | ---- | ----- |
| **[Monkey](https://developer.android.com/studio/test/other-testing-tools/monkey)** | Random exploration | Android's built-in event fuzzer; the baseline crash/robustness gate. |
| **[Appium](https://appium.io/)** | Scripted, cross-platform | WebDriver-based; drives Android and iOS from one script API. |
| **[Espresso](https://developer.android.com/training/testing/espresso)** / XCUITest | Scripted, native | In-process UI test frameworks for Android and iOS respectively. |
| **[Selenium](https://www.selenium.dev/)** / [Playwright](https://quality.stereobooster.com/playwright.md) | Scripted, web | WebDriver (the W3C standard) and its modern successor for browser UIs. |
| **[gremlins.js](https://github.com/marmelab/gremlins.js)** | Random exploration, web | In-browser monkey (click, scroll, type, forms); dependency-free but unreleased since 2020, typically run inside Playwright or Cypress. |
| **[Crawljax](https://github.com/crawljax/crawljax)** | Guided crawling, web | Event-driven state exploration for JavaScript apps; releases stalled since 2023. |

Device farms (Firebase Test Lab, AWS Device Farm, BrowserStack) are
commercial SaaS that host the physical-device matrix.

## When to use, when not

**Use:**

- **Automated exploration (start with Monkey)** as a cheap crash and ANR
  gate before release; it finds robustness bugs no scripted suite reaches.
- **Scripted example tests** for the functional flows that matter, since
  exploration cannot check correctness or pass input gates.
- **A combinatorial device matrix** on a device farm where fragmentation
  is a real risk.
- **[Visual snapshots](https://quality.stereobooster.com/snapshot-testing.md#visual-snapshot)** for
  UI-regression-sensitive surfaces.

**Don't:**

- **Expect exploration to verify behavior.** Its oracle is "did not
  crash"; correctness needs an oracle a person wrote.
- **Build a suite top-heavy with slow, flaky UI tests.** The
  [pyramid critique](https://quality.stereobooster.com/testing-folklore.md) is about
  exactly this failure mode.
- **Read a coverage plateau as thoroughness.** Automated exploration
  stalls at the first input gate, leaving the app's logic-heavy interior
  untouched.

## Evidence

- **Random exploration is a strong baseline**: as effective as the
  model-based and search-based Android tools on coverage and crash
  detection in a head-to-head comparison, and cheaper (Choudhary et al. 2015)[^choudhary2015].
- **Search-based exploration works at industrial scale**: Sapienz files
  short, actionable crash reports and ran on Facebook's apps
  (Mao et al. 2016; Alshahwan et al. 2018)[^mao2016] [^alshahwan2018].
- **Scope.** Automated GUI testing reduces the uncertainty that an app
  will crash or hang under real use; it says little about functional
  correctness, which still needs scripted oracles, and it cannot reach the
  states behind meaningful input. A systematic mapping of the field
  (Tramontana et al. 2019)[^tramontana2019] surveys the fuller landscape.

## Classification

- **Quality dimensions:** Reliability (crash and ANR robustness under unscripted event sequences), Functionality (functional correctness, via scripted example tests through the UI and visual snapshots), Performance (responsiveness; an ANR is where reliability meets performance) — a class of system under test whose one distinctive technique is automated GUI exploration (the axes below describe it). Scripted UI tests are example tests reached through a UI, visual checks are approval tests, and the device matrix is a combinatorial problem: details that route to existing methods rather than new ones.
- **Area:** Interactive GUI applications as the system under test, reached only through their interface: Android and iOS mobile apps (the research-rich core) and desktop and web front-ends. Event-driven, stateful, and fragmented across devices.
- **Guarantee:** Empirical — a search over a vast event space; a crash found is real, but coverage plateaus well short of the whole app and finding none proves nothing.

## Referenced by

- [Search-based software testing (SBST)](https://quality.stereobooster.com/search-based-software-testing.md) · Methods

## References

[^su2017]: Su, Ting, Guozhu Meng, Yuting Chen, et al. 2017. "[Guided, Stochastic Model-Based GUI Testing of Android Apps](https://tingsu.github.io/files/nasac2017-stoat.pdf)." *Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017)*, 245–56. <https://doi.org/10.1145/3106237.3106298>.
[^mao2016]: Mao, Ke, Mark Harman, and Yue Jia. 2016. "[Sapienz: Multi-objective Automated Testing for Android Applications](http://www0.cs.ucl.ac.uk/staff/Yue.Jia/resources/papers/MaoHJ2016.pdf)." *Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016)*, 94–105. <https://doi.org/10.1145/2931037.2931054>.
[^alshahwan2018]: Alshahwan, Nadia, Xinbo Gao, Mark Harman, et al. 2018. "[Deploying Search Based Software Engineering with Sapienz at Facebook](https://link.springer.com/content/pdf/10.1007%2F978-3-319-99241-9_1.pdf)." *Search-Based Software Engineering (SSBSE 2018)*, 3–45. [https://doi.org/10.1007/978-3-319-99241-9\\\_1](https://doi.org/10.1007/978-3-319-99241-9\_1).
[^choudhary2015]: Choudhary, Shauvik Roy, Alessandra Gorla, and Alessandro Orso. 2015. "[Automated Test Input Generation for Android: Are We There Yet?](https://software.imdea.org/~alessandra.gorla/papers/Choudhary-AndroidTestTools-ASE15.pdf)" *2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)*, 429–40. <https://doi.org/10.48550/arXiv.1503.07217>.
[^luo2014]: Luo, Qingzhou, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. "[An Empirical Analysis of Flaky Tests](https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf)." *Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '14)* (Hong Kong, China), 643–53. <https://doi.org/10.1145/2635868.2635920>.
[^tramontana2019]: Tramontana, Porfirio, Domenico Amalfitano, Nicola Amatucci, and Anna Rita Fasolino. 2019. "[Automated Functional Testing of Mobile Applications: A Systematic Mapping Study](https://link.springer.com/content/pdf/10.1007/s11219-018-9418-6.pdf)." *Software Quality Journal* 27 (1): 149–201. <https://doi.org/10.1007/s11219-018-9418-6>.

## Acronyms

- ANR — application not responding
- SBST — search-based software testing