# Data quality

Data-quality methods check the datasets a pipeline produces, such as tables,
files and partitions, on every run. The program and the data fail separately: an
upstream source changes a format, a feed stops, or a producer starts sending
nulls, and the pipeline's code has not changed at all. A check on the code passes
in each of these cases, so the check has to read the data itself, run after run.

Each method is classified on the
[data-quality characteristics](https://quality.stereobooster.com/characteristics.md) of ISO/IEC 25012, the
vocabulary data teams use, instead of the five software dimensions the other
methods on this site are classified on.

## The members

- **[Data-quality assertions](https://quality.stereobooster.com/data-quality-assertions.md)**: declared rules over
  each dataset (keys, nulls, references, row counts), checked as a gate before
  publication or as an alert after it.
- **[Data anomaly detection](https://quality.stereobooster.com/data-anomaly-detection.md)**: each batch's metrics
  compared with a band learned from the batches before it.
- **[Label-error detection](https://quality.stereobooster.com/label-error-detection.md)**: each label of a dataset
  that trains or evaluates a model, compared with a model's out-of-sample
  prediction for that example.
- **[Train-test leakage detection](https://quality.stereobooster.com/train-test-leakage.md)**: the test split
  of a dataset, searched for duplicates and near-duplicates of training
  examples and for groups that cross the split.
- **[Personal-data detection](https://quality.stereobooster.com/personal-data-detection.md)**: a dataset
  scanned with patterns and named-entity models for information that identifies
  a person.
- **[Duplicate-record detection](https://quality.stereobooster.com/duplicate-record-detection.md)**: records
  in one table, or across two, compared by their values to find those that
  describe the same entity under different keys.
- **[Data error detection](https://quality.stereobooster.com/data-error-detection.md)**: the values of a table
  scanned for those that break its patterns, its implicit rules or its
  distribution, without a rule set written in advance.
- **[Statistical integrity tests](https://quality.stereobooster.com/statistical-integrity-tests.md)**: a whole
  dataset's distribution compared with what genuine measurement would produce,
  to find fabricated, manipulated, clipped or rounded data.
- **[Training-corpus quality filtering](https://quality.stereobooster.com/training-corpus-filtering.md)**: the
  documents of a pretraining corpus kept or dropped by a score from rules, a
  classifier or an LLM.
- **[Annotation agreement](https://quality.stereobooster.com/annotation-agreement.md)**: the labels several
  annotators give the same items compared with each other and with items whose
  answer is known.

Two questions separate the members: what they check, and who writes the
oracle. Assertions and anomaly detection check each run's output. A person
declares an assertion, so it fires only on a violation of something known, and it
misses every failure nobody named. History supplies the anomaly band, so it
reports failures nobody anticipated, at the price of alarms on ordinary variation
and of accepting whatever defects the history already held. The other eight check
the contents of one dataset, usually before it is shared or used for training.
Label-error detection takes its oracle from a model trained on the rest of the
dataset, train-test leakage detection from a rule about the split (no test example
may also be in training), and personal-data detection from patterns and
named-entity models that recognize identifiers. Duplicate-record detection
compares records with each other, and a similarity threshold decides which pairs
are one entity. Data error detection infers what a column should look like from
the data, from other tables or from a few labeled cells, and flags the values
that do not fit. Statistical integrity tests take their oracle from the
distribution genuine data would have, and judge the dataset as a whole rather
than any one value. Training-corpus quality filtering has no correct answer to
compare with: a rule, a classifier or an LLM scores how useful a document is to
train on, and only models trained on the filtered data show whether the score
was right. Annotation agreement takes its oracle from the annotators themselves,
and measures whether the labeling can be reproduced rather than whether any
label is right.

## Methods and characteristics

| Method | Oracle | Guarantee | Characteristics checked |
| --- | --- | --- | --- |
| [Data-quality assertions](https://quality.stereobooster.com/data-quality-assertions.md) | Invariant | Empirical | Consistency, Completeness, Compliance |
| [Data anomaly detection](https://quality.stereobooster.com/data-anomaly-detection.md) | Aggregate | Empirical | Completeness, Currentness, Accuracy |
| [Label-error detection](https://quality.stereobooster.com/label-error-detection.md) | Statistical | Empirical | Accuracy |
| [Train-test leakage detection](https://quality.stereobooster.com/train-test-leakage.md) | Invariant | Empirical | Consistency |
| [Personal-data detection](https://quality.stereobooster.com/personal-data-detection.md) | Threshold | Empirical | Confidentiality |
| [Duplicate-record detection](https://quality.stereobooster.com/duplicate-record-detection.md) | Threshold | Empirical | Consistency, Accuracy |
| [Data error detection](https://quality.stereobooster.com/data-error-detection.md) | Statistical | Empirical | Accuracy, Consistency |
| [Statistical integrity tests](https://quality.stereobooster.com/statistical-integrity-tests.md) — Fabrication and manipulation | Statistical | Empirical | Credibility, Accuracy |
| [Statistical integrity tests](https://quality.stereobooster.com/statistical-integrity-tests.md) — [Digit preference and heaping](https://quality.stereobooster.com/statistical-integrity-tests.md#digit-preference-and-heaping) | Statistical | Empirical | Precision, Accuracy |
| [Training-corpus quality filtering](https://quality.stereobooster.com/training-corpus-filtering.md) — Rule filters | Threshold | Empirical | Understandability, Credibility |
| [Training-corpus quality filtering](https://quality.stereobooster.com/training-corpus-filtering.md) — Model-based filters | Statistical | Empirical | Understandability, Credibility |
| [Training-corpus quality filtering](https://quality.stereobooster.com/training-corpus-filtering.md) — LLM-judged filters | AI | Heuristic | Understandability, Credibility |
| [Annotation agreement](https://quality.stereobooster.com/annotation-agreement.md) — Agreement coefficient | Aggregate | Heuristic | Consistency |
| [Annotation agreement](https://quality.stereobooster.com/annotation-agreement.md) — Gold questions | Equality | Empirical | Accuracy |

## What the family catches

The family catches failures that enter with the data rather than the code: type
and format changes from a source, feeds that stop or arrive late, broken keys and
references, shifts in distribution, wrong labels, test examples copied from the
training data, personal information nobody meant to keep, one entity stored as
several records, wrong values no declared rule describes, data fabricated,
manipulated or clipped while every value stays plausible, values recorded more
coarsely than they claim, labels annotators cannot reproduce, and crawled
documents such as boilerplate and spam that are not worth training on.

No member checks whether a dataset represents the population it will be used
on. The count it would rest on, records per class or subgroup, is part of several
tools: [cleanlab](https://github.com/cleanlab/cleanlab) and [Deepchecks](https://github.com/deepchecks/deepchecks) flag a class
far rarer than the others, and
[TensorFlow Data Validation](https://github.com/tensorflow/data-validation) computes statistics per slice of the
data. The count is possible only for an attribute the dataset records.

The family does not catch a transformation that computes the wrong thing from good input
while its output stays plausible; that is a defect in the pipeline's code, found
by testing the code. Three program-side methods cover neighboring failures:
[schema and boundary validation](https://quality.stereobooster.com/schema-and-boundary-validation.md)
rejects a malformed record as it enters a program,
[database migration safety](https://quality.stereobooster.com/migration-safety.md) checks how a
schema change is applied to a live database, and
[schema compatibility checking](https://quality.stereobooster.com/schema-compatibility.md)
refuses a change to a table's published columns that its readers could not
handle. Testing the code that queries a database uses program-side methods with
SQL-specific tools: [SQL linters](https://quality.stereobooster.com/linters.md#sql) flag
anti-patterns in queries, [mutation](https://quality.stereobooster.com/mutation-testing.md#sql)
and [coverage](https://quality.stereobooster.com/coverage.md#sql) criteria for SQL measure whether a
test database exercises a query's conditions, and
[search-based generation](https://quality.stereobooster.com/search-based-software-testing.md#applications)
fills a test database that meets them. Comparing a changed pipeline's output tables with
the current version's, row by row, is a
[parallel run](https://quality.stereobooster.com/parallel-run.md#data-pipelines) over tables. For training and serving data,
[testing machine-learning systems](https://quality.stereobooster.com/testing-ml-systems.md) adds the
checks specific to models.

## Referenced by

- [Data-quality characteristics](https://quality.stereobooster.com/characteristics.md) · Data quality
- [Methods](https://quality.stereobooster.com/methods.md) · Methods