Skip to content

Software Quality

Parallel run

A parallel run is a comparison of two implementations on the same live request. The existing one is the control and the new one the candidate. The control's result is what the user gets, the candidate's is compared against it, and every mismatch is recorded. The control is the oracle, and the assertion is that the new version behaves like the old one on real production traffic — the input distribution no test author would think to fixture. A disagreement is a regression in the candidate, or occasionally a latent bug in the control the rewrite exposed.

The comparison takes one of two forms: in-process, wrapping two code paths in one service and comparing their results inline, or at the service boundary, in a comparison proxy that multicasts each request to both and diffs the responses.

A dark launch runs the new path and discards its result; a canary routes a fraction of traffic to it and watches aggregate metrics. Both limit blast radius and judge aggregate health; a parallel run compares the exact output of both paths on the same request.

What it catches

  • Behavioral regressions in a rewrite. The candidate disagrees with the control on real inputs, including the long-tail cases a hand-written suite misses.
  • Disagreement on production-only inputs. Malformed, oversized, or edge-case requests that never appear in fixtures.
  • Premature cutover. The mismatch rate is the go/no-go signal: a candidate that has agreed with the control across enough live traffic is ready to replace it.

A parallel run does not catch anything the two implementations get wrong in the same way — a shared dependency, a shared misreading of the spec — so like all differential testing it proves relative correctness, not absolute. It also checks only the paths traffic actually exercises.

Tools

  • In-process: GitHub's Scientist is the original, a Ruby gem with ports in ~15 languages (laboratory for Python, Scientist4J for Java, Scientist.net, and others).
  • At the service boundary: Diffy compares the candidate against two known-good instances, and treats whatever the two good ones disagree on as non-deterministic noise rather than a candidate diff.

Hazards

  • Candidate side effects. The candidate runs for real: a candidate that writes to a store, sends mail, or charges a card does damage. A parallel run assumes only the control's effects reach the world, so the candidate has to be pure or have its effects stubbed.
  • Non-determinism. Timestamps, ordering, and randomness produce spurious diffs; normalizing the outputs before comparison removes the diffs it can reach, and the rest need a tool that models the noise.
  • Latency and cost. Both paths execute on every compared request, so a service under load compares a sampled fraction of traffic.

Where it sits

Executing on production traffic makes a parallel run a continuous check, alongside contracts and runtime assertions and monitoring and observability. Its predicate comes from elsewhere: a second implementation supplies "matches the old implementation", while those methods check an invariant the author wrote by hand.

When to use, when not

Use it where a rewrite's correct output is hard to predict but the current behavior is trusted, and the candidate can run without side effects: optimizations, library migrations, language ports.

Don't reach for it when a cheaper oracle exists (a property or an example test catches the bug pre-merge, without production risk), when the candidate cannot be made side-effect-free, or when the question is aggregate health rather than per-request equivalence, which is a canary's job.

Evidence

No controlled study measures parallel run on its own. GitHub built Scientist to validate a risky rewrite against production traffic, and Twitter built Diffy for service migrations; both are engineering case studies, showing that the technique operates at scale rather than measuring how much it catches relative to the alternatives.

Differential oracle

These all answer the same question — does the candidate match a trusted reference? — and differ in how rigorously you compare, and in what the reference is. By rigour:

The reference itself varies too: a trusted implementation (a peer, the previous version, a gold-standard library), or a deliberately simple executable spec authored to be the reference.

Classification

  • Quality dimensions: Functionality, Maintainability (the technique exists to make a refactor or rewrite safe).
  • Area: Refactors and reimplementations verified behind live traffic — library migrations, language ports, slow-path → fast-path optimizations — with the existing code as the reference.
  • Guarantee: Empirical — agreement across the real input distribution as it arrives, never a proof.

Referenced by