Data-quality methods check the datasets a pipeline produces, such as tables, files and partitions, on every run. The program and the data fail separately: an upstream source changes a format, a feed stops, or a producer starts sending nulls, and the pipeline's code has not changed at all. A check on the code passes in each of these cases, so the check has to read the data itself, run after run.
Each method is classified on the data-quality characteristics of ISO/IEC 25012, the vocabulary data teams use, instead of the five software dimensions the other methods on this site are classified on.
The members¶
- Data-quality assertions: declared rules over each dataset (keys, nulls, references, row counts), checked as a gate before publication or as an alert after it.
- Data anomaly detection: each batch's metrics compared with a band learned from the batches before it.
- Label-error detection: each label of a dataset that trains or evaluates a model, compared with a model's out-of-sample prediction for that example.
- Train-test leakage detection: the test split of a dataset, searched for duplicates and near-duplicates of training examples and for groups that cross the split.
- Personal-data detection: a dataset scanned with patterns and named-entity models for information that identifies a person.
- Duplicate-record detection: records in one table, or across two, compared by their values to find those that describe the same entity under different keys.
- Data error detection: the values of a table scanned for those that break its patterns, its implicit rules or its distribution, without a rule set written in advance.
- Statistical integrity tests: a whole dataset's distribution compared with what genuine measurement would produce, to find fabricated, manipulated, clipped or rounded data.
- Training-corpus quality filtering: the documents of a pretraining corpus kept or dropped by a score from rules, a classifier or an LLM.
- Annotation agreement: the labels several annotators give the same items compared with each other and with items whose answer is known.
Two questions separate the members: what they check, and who writes the oracle. Assertions and anomaly detection check each run's output. A person declares an assertion, so it fires only on a violation of something known, and it misses every failure nobody named. History supplies the anomaly band, so it reports failures nobody anticipated, at the price of alarms on ordinary variation and of accepting whatever defects the history already held. The other eight check the contents of one dataset, usually before it is shared or used for training. Label-error detection takes its oracle from a model trained on the rest of the dataset, train-test leakage detection from a rule about the split (no test example may also be in training), and personal-data detection from patterns and named-entity models that recognize identifiers. Duplicate-record detection compares records with each other, and a similarity threshold decides which pairs are one entity. Data error detection infers what a column should look like from the data, from other tables or from a few labeled cells, and flags the values that do not fit. Statistical integrity tests take their oracle from the distribution genuine data would have, and judge the dataset as a whole rather than any one value. Training-corpus quality filtering has no correct answer to compare with: a rule, a classifier or an LLM scores how useful a document is to train on, and only models trained on the filtered data show whether the score was right. Annotation agreement takes its oracle from the annotators themselves, and measures whether the labeling can be reproduced rather than whether any label is right.
Methods and characteristics¶
| Method | Oracle | Guarantee | Characteristics checked |
|---|---|---|---|
| Data-quality assertions | Invariant | Empirical | Consistency, Completeness, Compliance |
| Data anomaly detection | Aggregate | Empirical | Completeness, Currentness, Accuracy |
| Label-error detection | Statistical | Empirical | Accuracy |
| Train-test leakage detection | Invariant | Empirical | Consistency |
| Personal-data detection | Threshold | Empirical | Confidentiality |
| Duplicate-record detection | Threshold | Empirical | Consistency, Accuracy |
| Data error detection | Statistical | Empirical | Accuracy, Consistency |
| Statistical integrity tests — Fabrication and manipulation | Statistical | Empirical | Credibility, Accuracy |
| Statistical integrity tests — Digit preference and heaping | Statistical | Empirical | Precision, Accuracy |
| Training-corpus quality filtering — Rule filters | Threshold | Empirical | Understandability, Credibility |
| Training-corpus quality filtering — Model-based filters | Statistical | Empirical | Understandability, Credibility |
| Training-corpus quality filtering — LLM-judged filters | AI | Heuristic | Understandability, Credibility |
| Annotation agreement — Agreement coefficient | Aggregate | Heuristic | Consistency |
| Annotation agreement — Gold questions | Equality | Empirical | Accuracy |
What the family catches¶
The family catches failures that enter with the data rather than the code: type and format changes from a source, feeds that stop or arrive late, broken keys and references, shifts in distribution, wrong labels, test examples copied from the training data, personal information nobody meant to keep, one entity stored as several records, wrong values no declared rule describes, data fabricated, manipulated or clipped while every value stays plausible, values recorded more coarsely than they claim, labels annotators cannot reproduce, and crawled documents such as boilerplate and spam that are not worth training on.
No member checks whether a dataset represents the population it will be used on. The count it would rest on, records per class or subgroup, is part of several tools: cleanlab and Deepchecks flag a class far rarer than the others, and TensorFlow Data Validation computes statistics per slice of the data. The count is possible only for an attribute the dataset records.
The family does not catch a transformation that computes the wrong thing from good input while its output stays plausible; that is a defect in the pipeline's code, found by testing the code. Three program-side methods cover neighboring failures: schema and boundary validation rejects a malformed record as it enters a program, database migration safety checks how a schema change is applied to a live database, and schema compatibility checking refuses a change to a table's published columns that its readers could not handle. Testing the code that queries a database uses program-side methods with SQL-specific tools: SQL linters flag anti-patterns in queries, mutation and coverage criteria for SQL measure whether a test database exercises a query's conditions, and search-based generation fills a test database that meets them. Comparing a changed pipeline's output tables with the current version's, row by row, is a parallel run over tables. For training and serving data, testing machine-learning systems adds the checks specific to models.
Referenced by¶
- Data-quality characteristics · Data quality
- Methods · Methods