Skip to content

Software Quality

Snapshot and approval testing

A snapshot test, also called an approval test, compares a run's output against a stored capture of that same output taken earlier. The first run has nothing to compare against, so it records what the code produced; a person reads that recording and approves it. The capture is committed next to the code, and when a diff appears someone reads it and decides — approve the new output, or fix the code that changed it.

A snapshot test sits inside an example test: the input is fixed and hand-picked, the guarantee is empirical, and an approved value is one of the assertions an example test may carry. What separates them is the provenance of the expected value — an example test's assertion is authored, a snapshot's is captured from a run.

The captured artifact is what varies: a serialized value or block of text, a rendered image compared under a tolerance, and a recorded interaction at a service seam replayed against the side under test.

What it catches

  • Unintended change in a wide output. One assertion covers a whole serialized structure, including the fields nobody thought to assert on.
  • Behavior no specification describes. Legacy code has an observable output and no written statement of what it should be.
  • Reshapes that were supposed to change nothing. Extract a function, swap a library, reorder a pipeline: if the captured output moves, the reshape was not behavior-preserving.
  • Rendering regressions. These show up in the pixels and nowhere else.
  • Drift at a seam. A response that gains a field, renames one, or changes a status code no longer matches the recording.
  • Diagnostic text. Error messages, log lines, and CLI help are output whose exact wording matters and which few people assert on by hand.

What a snapshot cannot catch

A bug that exists before the first approval is not one the test can find, because the test's expected value is that bug. A bug introduced later becomes the new reference the moment someone approves the diff.

A snapshot also sees only what the serializer emits, so behavior outside that projection — a side effect, a database write, a timing property — is unverified no matter how large the captured file grows.

Value and text snapshots

The captured artifact is a serialized value: a rendered component tree, a JSON payload, a formatted table, the output of a parser. Snapshots live either in a separate file keyed by test name, or inline, written back into the test source by the runner, which keeps the expected value in view during review at the cost of churn in the test file.

Granularity: a capture of a whole page fails on any change anywhere in it and forces a reviewer to locate the relevant difference, while a capture per component fails narrowly and reads like an assertion. Serializer stability: the pinned artifact is the serialization, not the value, so a formatter upgrade or a custom serializer edit invalidates every capture at once.

Anything nondeterministic inside the captured output moves the diff without any code change: timestamps, generated identifiers, hash-ordered keys, absolute paths.

Pinning legacy behavior before a reshape

A characterization test is an approval test pointed at code whose intended behavior was never written down (Feathers 2004)1. This is the one use where approving unexamined output is defensible.

Visual snapshots

The captured artifact is an image. A test drives the interface to a chosen state, takes a screenshot, and compares it against the approved image. For a user interface the render is the behavior: a broken layout, a color regression, a clipped label, and a font fallback are all invisible to assertions on the DOM.

A render depends on the machine that produced it, so the comparison only checks the code rather than the machine if the environment is pinned: one browser build, one viewport, one device scale factor, one set of fonts, captured in a container image.

The tolerance, and what it hides

Screenshot comparison is equality with slack, expressed as a share of differing pixels or a per-pixel color delta. Raising the tolerance hides real regressions underneath it; lowering it lets the render environment decide verdicts. A pinned environment is what lets the tolerance stay small.

A pixel-unit diff reports every changed pixel, so a change that moves an element rather than altering it flags the shifted region and everything downstream of it, and the differences that matter arrive buried under the shift (Tanno et al. 2020)2. The mitigation is a narrower capture.

Recorded interactions at a seam

The captured artifact is an interaction: a request and the response it drew, recorded once from a real exchange and replayed afterwards. Pointed at a service boundary, this is how a test exercises one side of a seam without the other side running. The seam need not be a network call: recording the protocol and values a client exchanges with a library, then diffing that recording after an upgrade, applies the same mechanism to a dependency boundary (Monce et al. 2025)3.

Contract testing in the consumer-driven sense (Pact and its relatives) is this technique applied across independently deployed services. The consumer records the requests it issues and the responses it expects; the provider is later replayed against those recordings in isolation. It is unrelated to design by contract, where a contract is a runtime predicate inside one component.

What Pact adds beyond a snapshot is a broker that turns one team's recorded expectations into another team's replay obligations and gates deploys on them. In a monorepo with one build graph there is nothing to broker: provider and consumer change in a single commit, and a breaking change fails the shared build at head.

Live traffic can stand in for the recording: a service mesh sidecar mirrors production requests to a candidate version, and a comparator diffs the candidate against the running implementation (Envoy does the mirroring, Diffy the comparison), swapping the curated recording for a differential oracle. Neither arrangement removes deployment skew: the two sides roll out at different moments, so some check must guard compatibility across the rollout window — brokered snapshots, schema-compatibility rules (protobuf back- and forward-compatibility), or live differential.

The two roles of one recording

A recording replayed at a seam does two jobs at once. Replaying the recorded response stubs out the counterpart and supplies input. Matching the outgoing request against the recorded request is an equality check against a captured expected value: change the URL, method, or body and the match fails.

Matching strictly on method, path, and body makes the recording a genuine check on outbound behavior; loosening the match to method and host, permitting playback repeats, or recording new episodes on the fly weakens it until nothing is being checked: the recording is then a test double, and the surrounding test's own assertions are its oracle.

Replay also removes the network instead of verifying under it. Timeouts, retries, reordering, and partial failure are absent from a replayed run by construction, so they need fault injection, chaos engineering, or deterministic simulation testing rather than a larger set of recordings.

Approving a change

The approval is the method. The failure mode comes from the runner itself: it offers to rewrite the expected value, rewriting it is one flag, and doing so converts a failing test into a passing test without anyone establishing which output was correct.

A capture small enough for a reviewer to hold the whole diff in mind is what makes the approval a decision. In code review the snapshot diff is part of the change, not noise attached to it.

An AI agent can do the work around this step and not the step itself. It can write the capture call, run the suite, read a red diff, and classify what changed: on a labeled set of iOS snapshot failures a vision-language model sorted failure causes at 84% recall, though prompting it to selectively ignore intended changes did not work (Kaynak et al. 2025)4. It cannot approve, because approval is the judgment that some output is the one that ought to be produced.

Tools

  • JavaScript / TypeScript — Vitest and Jest ship toMatchSnapshot and toMatchInlineSnapshot.
  • Python — syrupy, a pytest plugin.
  • Rust — insta, with a review command for accepting pending captures one at a time.
  • .NET, Java, C++, and others — Verify and ApprovalTests.
  • Swift / iOS — swift-snapshot-testing, which captures values and rendered views.
  • Go — no dominant library; the idiomatic pattern is a golden file next to the test and an -update flag that rewrites it.

For rendered output:

  • In-process screenshot comparison — Playwright's toHaveScreenshot; jest-image-snapshot for the Jest and Vitest matcher form.
  • Android — Paparazzi renders views on the JVM, so a visual capture needs no emulator or device.
  • Comparators — odiff and pixelmatch, when the comparison is separated from the runner.
  • Baseline storage and diff reports — reg-suit keeps approved images outside the repository and posts a review report per change.
  • State enumeration — Storybook, where each story is a pinned component state a capture can be taken from.

For recorded interactions:

  • Cassette libraries — VCR (Ruby), vcrpy, nock (JavaScript).
  • Standalone recorders and stub servers — WireMock and mountebank, both able to record a live exchange and replay it.
  • Consumer-driven contracts — Pact, which adds the consumer/provider protocol and the broker over the same recording.

When not to use

  • The question is whether the output is right — write the expected value by hand, as an example test, where authoring it forces the decision.
  • The output cannot be made deterministic — a clock you cannot freeze, a live network, genuine concurrency. The capture then fails on runs that changed nothing; use a property or a statistical oracle instead of a pinned value.
  • A reference implementation exists. Diffing against it is differential testing, which checks relative correctness rather than stability.
  • The property is structural and expressible directly — a schema, an invariant, a type. A predicate oracle states the constraint once and holds across inputs, where a capture pins one output.
  • Nobody will read the diffs. The update flag then decides every verdict.

Evidence

How much a snapshot suite benefits developers is still unclear: a 2023 mining study of the technique states that gap directly and lists it as future work (Fujita et al. 2023)5.

Four participants compared two screens side by side, each screen carrying eleven changes, and overlooked six differences in all (Tanno et al. 2020)2; the task was manual review rather than review of a computed diff.

Across independently deployed services the empirical base is thin: a 2025 systematic review of consumer-driven contract testing describes its own contribution as rare empirical data and rests its positive findings on a single action-research study (Schwarz et al. 2025)6.

Same name, different thing — Design by Contract vs contract testing (Pact)

These share only the word contract. A contract here is a runtime predicate inside one component — a precondition, postcondition, or invariant. Contract testing (Pact, consumer-driven) is an integration check at a service boundary: it records the messages a consumer expects and replays them against the provider. Different method, different failure — a broken assertion inside a program versus two services drifting out of agreement.

Classification

  • Quality dimensions: Functionality, Maintainability: modifiability — each instance pins behavior and makes a reshape visible; for a rendered interface the captured output is the behavior, and at a seam the pinned recording is what keeps independently deployed versions compatible.
  • Area: Output with a stable serialization — rendered component trees, formatted reports, compiler and CLI output, diagnostic text, wire payloads; legacy code with no written specification, pinned before a reshape; service seams whose recorded interactions are replayed against the side under test.
  • Guarantee: Empirical.

Referenced by

References

  1. Feathers, Michael. 2004. Working Effectively with Legacy Code. Prentice Hall PTR. ↩

  2. Tanno, Haruto, Yu Adachi, Yu Yoshimura, Katsuyuki Natsukawa, and Hideya Iwasaki. 2020. "Region-based Detection of Essential Differences in Image-based Visual Regression Testing." Journal of Information Processing 28: 268–78. https://doi.org/10.2197/ipsjjip.28.268. ↩↩

  3. Monce, Gustave, Thomas Degueule, Jean-Rémy Falleri, and Romain Robbes. 2025. "Client–Library Compatibility Testing with API Interaction Snapshots." IEEE International Conference on Software Maintenance and Evolution (ICSME), 791–96. https://doi.org/10.1109/ICSME64153.2025.00081. ↩

  4. Kaynak, Ergün Batuhan, Mayasah Lami, Sahand Moslemi, and Anil Koyuncu. 2025. "LLMShot: Reducing Snapshot Testing Maintenance via LLMs." IEEE International Conference on Software Maintenance and Evolution (ICSME), 827–32. https://doi.org/10.1109/ICSME64153.2025.00087. ↩

  5. Fujita, Shun, Yutaro Kashiwa, Bin Lin, and Hajimu Iida. 2023. "An Empirical Study on the Use of Snapshot Testing." IEEE International Conference on Software Maintenance and Evolution (ICSME), 335–40. https://doi.org/10.1109/ICSME58846.2023.00041. ↩

  6. Schwarz, Georg-Daniel, Felix Quast, and Dirk Riehle. 2025. "Ensuring Syntactic Interoperability Using Consumer-Driven Contract Testing." Software Testing, Verification and Reliability 35 (5). https://doi.org/10.1002/stvr.70006. ↩