Skip to content

Software Quality

Train-test leakage detection

Finding the test examples a model could have memorized because they, or their near-copies, are also in its training data.

Train-test leakage detection checks that the data a model is evaluated on is independent of the data it was trained on. A held-out test set estimates how a model performs on examples it has never seen, and the estimate holds only if no test example, or a near-copy of one, is also in the training set. A model scores well on such an example by remembering it, so the leaked share of the test set measures memorization and reports it as generalization.

How overlap is found

  • Exact duplicates. Each example is normalized and hashed, and a test example whose hash occurs in the training set is flagged.
  • Near-duplicates. Examples that differ in whitespace, comments, resizing or a few words have different hashes. Tools compare them by distance instead: nearest neighbors in a model's feature space for images (Barz and Denzler 2020)1, clone detection for source code (Allamanis 2019)2, shared substrings and approximate matching for text (Lee et al. 2022)3. A pair closer than a threshold is a candidate, and the threshold trades missed copies against flagged look-alikes.
  • Groups and time. Examples drawn from one document, patient or project are related even when no two are duplicates, and so are a record and its later revision. A random split scatters them across training and test; the check is that each group, and each period, falls on one side of the split.

Leakage also enters through the pipeline's code: a scaler or vectorizer fitted on the whole dataset before the split carries test statistics into training, and a test set used to choose between models becomes part of the training process. That form is visible in the code, not the data, and a static analysis of the notebook finds it (Yang et al. 2022)4.

What it catches

  • Test examples copied from training. Exact and near-duplicate examples on both sides of a split, including copies that survived the removal of exact duplicates.
  • Related examples split apart. Sentences of one document, scans of one patient or files of one project, divided between training and test.
  • Future data in training. Records dated after the test period in a split meant to predict forward in time.
  • Preprocessing fitted on test data, when the check reads the pipeline's code.

What it does not catch:

  • a feature that encodes the label, such as an identifier assigned after the outcome was known, which a duplicate search never compares;
  • a duplicate the similarity measure does not recognize, such as a paraphrase or a translated copy;
  • a test set used repeatedly to choose models, unless that happens in the code the analysis reads.

Tools

  • Deepchecks — Python library of validation checks; Train Test Samples Mix reports the share of test samples that also occur in the training set, and further checks look for date and index leakage between the splits.
  • cleanlab — its Datalab finds near-duplicate sets within a dataset from features or a nearest-neighbor graph, and tests whether the data's order violates the assumption that examples are independent and identically distributed.
  • deduplicate-text-datasets — the exact-substring deduplicator released with the study of language-model training data, including a command that finds substrings shared between two datasets.
  • leakage-analysis — research implementation of a static analysis that detects preprocessing, overlap and multi-test leakage in Python notebooks.

When to use, when not

Use:

  • On a benchmark before models are ranked on it, since a model that memorizes more scores better on the leaked share.
  • On data mined from the web or from code hosts, where copies of the same content are common.
  • When drawing a split from records that share a source, such as one patient, user or repository, so the split is made by group.

Don't:

  • To remove every near-duplicate from a test set without review. A threshold loose enough to catch every copy also flags distinct examples that look alike.
  • As proof that a test score will hold in production. A test set free of training copies can still differ from the data the model will meet.

Evidence

  • Benchmark test sets contain training images, and memorized ones are scored almost perfectly. Searching CIFAR-10 and CIFAR-100 in a network's feature space, with manual review, found 286 and 891 test images with near-duplicates in the training set; even a simple ResNet-110 classified almost all of them correctly. With the duplicates replaced, error rates rose on average by 0.41 percentage points on CIFAR-10 and 2.73 on CIFAR-100, by up to 12% relative, while the ranking of architectures did not change on CIFAR-100 and two architectures swapped places on CIFAR-10 (Barz and Denzler 2020)1.
  • Duplication in code corpora inflates results. Ten public datasets used for machine learning on code still held many near-duplicate files after their authors removed exact clones, and reported metrics were sometimes inflated by up to 100% when testing on duplicated corpora rather than de-duplicated ones (Allamanis 2019)2.
  • Web text overlaps its own validation sets. In language-model corpora, 4.6% of the C4 validation set and 14.4% of the RealNews validation set had an approximate duplicate in the training set (Lee et al. 2022)3.
  • Splitting related examples apart raises scores. In the AIMed relation-extraction dataset, keeping each document in only one fold dropped F-scores by more than 10 points across three models (Elangovan et al. 2021)5.
  • Leakage misleads whole research fields. A survey of reviews found leakage in 17 fields, affecting 294 papers. In civil war prediction, complex models believed to outperform logistic regression did not perform substantively better once the errors were corrected (Kapoor and Narayanan 2023)6.
  • Leakage in pipeline code is common and statically detectable. A static analysis of 107,603 GitHub notebooks found leakage in 29.6% of them, at a precision of 97.6% and a recall of 67.8% on a manually labeled sample (Yang et al. 2022)4.

Classification

  • Data-quality characteristics: Consistency (a test split is coherent with its training split only when no example or group appears in both; a leak inflates the test score).
  • Area: Datasets split into training, validation and test sets for a model: benchmark datasets, corpora mined from code and the web, and the splits a team draws from its own records.
  • Guarantee: Empirical — an empty overlap shows that no pair crossed the split under the tool's similarity measure; a duplicate the measure does not recognize, and leakage through features or preprocessing, pass.

Referenced by

References

  1. Barz, Björn, and Joachim Denzler. 2020. "Do We Train on Test Data? Purging CIFAR of Near-Duplicates." Journal of Imaging 6 (6): 41. https://doi.org/10.3390/jimaging6060041. ↩↩

  2. Allamanis, Miltiadis. 2019. "The Adverse Effects of Code Duplication in Machine Learning Models of Code." Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2019). https://doi.org/10.1145/3359591.3359735. ↩↩

  3. Lee, Katherine, Daphne Ippolito, Andrew Nystrom, et al. 2022. "Deduplicating Training Data Makes Language Models Better." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2022.acl-long.577. ↩↩

  4. Yang, Chenyang, Rachel A. Brower-Sinning, Grace A. Lewis, and Christian Kästner. 2022. "Data Leakage in Notebooks: Static Detection and Better Processes." Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022). https://doi.org/10.1145/3551349.3556918. ↩↩

  5. Elangovan, Aparna, Jiayuan He, and Karin Verspoor. 2021. "Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021). https://doi.org/10.18653/v1/2021.eacl-main.113. ↩

  6. Kapoor, Sayash, and Arvind Narayanan. 2023. "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns 4 (9): 100804. https://doi.org/10.1016/j.patter.2023.100804. ↩