# Data anomaly detection Data anomaly detection compares each new batch a pipeline produces with the batches before it, and flags the batch whose quality metrics fall outside the range its history predicts. On each run, a tool computes metrics over the new batch (row count, fraction of nulls per column, distinct values, quantiles, time of the newest record) and adds them to a time series. A pipeline whose every past run produced 50 rows, one per US state, has made 50 its expected count, and a run producing 10 rows stands out. The rule is learned rather than written. Some tools fit a band around each metric separately. Others train a novelty-detection model on the vector of metrics from past batches and ask whether the new one resembles them (Redyuk et al. 2021)[^redyuk2021]. Because the threshold is recomputed from history on every run, the method reports errors nobody anticipated, which a [data-quality assertion](https://quality.stereobooster.com/data-quality-assertions.md) cannot, since an assertion checks only what someone named. A learned band treats past batches as correct: Auto-Validate by-History assumes them free of quality issues, on the grounds that engineers check a pipeline's first runs by hand (Tu et al. 2023)[^tu2023]. A defect present from the first run is part of the normal range and is never flagged. ## What it catches - **Volume drops and spikes.** An upstream job that fails silently leaves a batch with a fraction of its usual rows, or none. - **Late data.** A table whose newest record is older than its update pattern implies has fallen behind its source. - **Corrupted columns.** The null fraction or distinct count of a column jumps after an upstream change breaks how the column is filled. - **Distribution shift.** Values can stay inside every declared range while their mix changes, such as one category's share doubling. What anomaly detection does **not** catch: - a defect the history already contained; - a wrong value that moves no tracked metric; - a few bad rows in a large batch, which aggregate metrics dilute below the band. ## False alarms decide adoption A band tight enough to catch a small corruption also fires on ordinary variation, and the count multiplies across columns: a 1% per-column false-alarm rate over 10,000 monitored columns is about a hundred alerts per run. Research systems attack the rate directly. Auto-Validate by-History chooses each pipeline's rules so that their false-positive rate stays under a chosen target, such as 0.1% (Tu et al. 2023)[^tu2023]. A validation system built at Meta clusters correlated features and monitors cluster-wide statistics instead of alerting on each feature (Shankar et al. 2023)[^shankar2023]. ## Tools - **[Elementary](https://github.com/elementary-data/elementary)** — open-source [dbt](https://github.com/dbt-labs/dbt-core) package; anomaly tests such as `volume_anomalies` compare each time bucket with a training window of earlier buckets (14 days by default). - **[Deequ](https://github.com/awslabs/deequ)** — stores each run's metrics in a repository and runs anomaly-detection strategies over their history. - **[TensorFlow Data Validation](https://github.com/tensorflow/data-validation)** — compares a new span's statistics with the previous span's (drift) or serving data with training data (skew) under a distance threshold; built for ML pipelines. - **Monte Carlo** (commercial) — hosted data-observability service that learns freshness, volume and schema monitors for warehouse tables. ## When to use, when not **Use:** - On recurring pipelines with enough history to learn from, such as weeks of daily runs. - Across many tables at once, where writing rules for every column is not practical. - Alongside [data-quality assertions](https://quality.stereobooster.com/data-quality-assertions.md): rules for what is known, anomaly detection for what is not. **Don't:** - On a new pipeline. With no history there is no band, and a band learned from unchecked history accepts its defects. - With every alert routed to a person before the false-alarm rate is known. Engineers who learn that alerts are usually false stop reading them. - On runs that change by design, such as a backfill or a migration, where history does not predict the output. ## Evidence - **Learned validation can match hand-tuned rules.** On five real-world datasets with real and synthetic errors, novelty detection over batch statistics outperformed other automated solutions and reached the quality of baselines hand-tuned with domain expertise (Redyuk et al. 2021)[^redyuk2021]. - **A fixed threshold was abandoned at Meta.** There, an alert on a 30% drop in a feature's completeness produced so many false positives that ML engineers silenced it and then abandoned the method; the clustering system the team then built gave a 2.1× average improvement in precision over the baseline on Instagram data (Shankar et al. 2023)[^shankar2023]. - **History-based rules beat generic detectors at scale.** On 2,000 production pipelines at Microsoft, Auto-Validate by-History outperformed commercial solutions and anomaly-detection methods from the machine-learning and database literature (Tu et al. 2023)[^tu2023]. ## Classification - **Data-quality characteristics:** Completeness (a feed that stops or shrinks, or a column whose null fraction jumps, leaves records or values missing), Currentness (data that arrives later than its history predicts is older than its consumers expect), Accuracy (a corrupted batch shifts the distribution of the columns it damaged). - **Area:** Recurring pipelines — daily or hourly loads, ingested batches, feature tables — where each new batch can be compared with the history of earlier ones. - **Guarantee:** Empirical — a quiet run shows the batch resembled its history on the metrics tracked; an error history already contained, or one that moves no tracked metric, passes. ## Referenced by - [Data quality](https://quality.stereobooster.com/data-quality.md) · Data quality - [Data-quality assertions](https://quality.stereobooster.com/data-quality-assertions.md) · Data quality - [Testing machine-learning systems](https://quality.stereobooster.com/testing-ml-systems.md) · Methods ## References [^redyuk2021]: Redyuk, Sergey, Zoi Kaoudi, Volker Markl, and Sebastian Schelter. 2021. "[Automating Data Quality Validation for Dynamic Data Ingestion](https://openproceedings.org/2021/conf/edbt/p79.pdf)." *Proceedings of the 24th International Conference on Extending Database Technology (EDBT 2021)*. . [^tu2023]: Tu, Dezhan, Yeye He, Weiwei Cui, et al. 2023. "[Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines](https://arxiv.org/pdf/2306.02421)." *Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '23)*. . [^shankar2023]: Shankar, Shreya, Labib Fawaz, Karl Gyllstrom, and Aditya G. Parameswaran. 2023. *[Moving Fast With Broken Data](https://arxiv.org/pdf/2303.06094.pdf)*. .