Skip to content

Software Quality

Training-corpus quality filtering

Scoring every document of a web-scale corpus and dropping those below a threshold before a model trains on it.

Training-corpus quality filtering scores every document of a corpus collected in bulk, such as a crawl of the web, and drops the documents that score below a threshold before a language model trains on the corpus. It removes boilerplate, spam, text in other languages and fragments that are not prose. No document has a correct score to compare with: the score stands in for how useful the document is to train on, and whether it does is measured only by training models on the filtered data and comparing them.

Kinds of filter

  • Rule filters {#rule-filters} compute statistics of each document, such as its length, its mean word length, the share of its lines that end in punctuation or repeat, and whether it contains a word from a blocklist, and drop documents outside fixed bounds. The rule sets of the C4 and Gopher corpora are the common starting points.
  • Model-based filters {#model-based-filters} score each document with a classifier trained to separate reference text, such as Wikipedia or curated sources, from raw crawl, or with the perplexity of a language model trained on reference text. The documents above a percentile of the score are kept.
  • LLM-judged filters {#llm-judged-filters} have an LLM rate a sample of documents on a stated criterion, such as educational value. Running the LLM on every document costs too much, so a small classifier trained on its ratings scores the rest of the corpus.

Pipelines identify each document's language first and drop documents outside the target language, and they deduplicate the corpus in a separate step. Removal of benchmark text belongs to train-test leakage detection, and removal of personal information to personal-data detection.

What it catches

  • Boilerplate. Navigation menus, cookie notices and terms of use extracted with a page's text.
  • Spam and generated text. Keyword lists, link farms and text produced by templates.
  • Fragments. Lists, tables and code that a prose filter treats as noise.
  • Text in another language. Documents the target language's model should not see.

What it does not catch:

  • false statements written in fluent prose, which every filter scores as text worth training on;
  • duplicate documents, which deduplication removes;
  • benchmark text, which train-test leakage detection finds.

Tools

  • datatrove — Python library for processing text corpora, used to build FineWeb; implements the C4, Gopher and FineWeb rule filters, language identification and classifier filters.
  • NeMo Curator — NVIDIA's open-source corpus pipeline; rule filters, classifier filters and GPU classifiers, among them one for educational value.
  • Data-Juicer — Python data-processing system with more than a hundred operators, including perplexity filters and filters scored by an LLM.
  • Dolma toolkit — the Allen Institute for AI's toolkit, which tags documents with C4 and Gopher rules, language, toxicity and personal information, then keeps documents by tag thresholds.

When to use, when not

Use:

  • Before pretraining on a crawl, where most raw documents are not prose a model should learn from.
  • With thresholds chosen by training small models on the filtered data, since a filter's agreement with people's judgment of quality does not predict which filtered data trains the better model.

Don't:

  • Without auditing what a blocklist or a classifier trained on formal reference text removes: both drop text in minority dialects and text from less advantaged authors at higher rates.
  • As a check of correctness, since a filter judges whether a document reads like good training text, not whether what it says is true.

Evidence

  • Removing C4's filters made a model worse on every task. A T5 model pretrained on a version of C4 without its rule filters performed worst of the compared corpora on every task (Raffel et al. 2020)1.
  • Rule filters needed tuning per source; deduplication did not. Applied to other web corpora, the same rule filters needed source-dependent tuning, while stringent deduplication improved zero-shot performance consistently (Penedo et al. 2023)2.
  • A classifier trained on LLM ratings raised benchmark scores. Llama-3-70B-Instruct rated 460,000 web pages for educational value, and a classifier trained on its ratings filtered the FineWeb corpus; a model trained on the result scored 37% on MMLU, against 33% for one trained on FineWeb (Penedo et al. 2024)3.
  • A simple classifier beat an LLM judge, and agreement with people predicted nothing. At the same scale, a fastText classifier filter scored 30.2 on the DataComp-LM core benchmark, perplexity filtering 29.0 and AskLLM, which asks an LLM whether a document is useful, 28.6. AskLLM agreed with human quality labels more closely than the fastText filters, and across filters that agreement did not predict the trained model's score (Li et al. 2024)4.
  • Quality filtering helped most tasks, unpredictably. A classifier quality filter improved downstream performance across the tasks tested despite removing training data, but its benefit could not be predicted from characteristics of the text, and it also increased toxic generation (Longpre et al. 2024)5.
  • LLM ratings matched a 50% longer training run. Sampling training data by an LLM's pairwise judgments of educational value gave a model comparable to one trained on uniformly sampled data for 50% more steps (Wettig et al. 2024)6.
  • A blocklist removed minority dialects. C4's blocklist removed 42% of documents in African American English and 32% in Hispanic-aligned English, against 6.2% in white-aligned English (Dodge et al. 2021)7.
  • A quality classifier favored wealthier authors. A replication of GPT-3's quality filter rated newspapers of larger schools in wealthier, more educated and urban areas as higher quality, and its scores did not align with factuality ratings or literary awards (Gururangan et al. 2022)8.

Classification

  • Data-quality characteristics: Understandability (text in the wrong language, boilerplate and fragments are not prose a model can learn to read), Credibility (spam and machine-generated text are not text a model should learn to believe).
  • Area: Corpora collected in bulk to pretrain language models — crawls of the web, scraped books and code — where most raw documents are not text a model should learn from.
  • Guarantee: Empirical, Heuristic (LLM-judged filters).

Referenced by

References

  1. Raffel, Colin, Noam Shazeer, Adam Roberts, et al. 2020. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." Journal of Machine Learning Research 21 (140). https://jmlr.org/papers/volume21/20-074/20-074.pdf. ↩

  2. Penedo, Guilherme, Quentin Malartic, Daniel Hesslow, et al. 2023. "The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only." Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://doi.org/10.52202/075280-3464. ↩

  3. Penedo, Guilherme, Hynek Kydlíček, Loubna Ben Allal, et al. 2024. "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale." Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://doi.org/10.52202/079017-0970. ↩

  4. Li, Jeffrey, Alex Fang, Georgios Smyrnis, et al. 2024. "DataComp-LM: In Search of the Next Generation of Training Sets for Language Models." Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://doi.org/10.52202/079017-0455. ↩

  5. Longpre, Shayne, Gregory Yauney, Emily Reif, et al. 2024. "A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). https://doi.org/10.18653/v1/2024.naacl-long.179. ↩

  6. Wettig, Alexander, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. "QuRating: Selecting High-Quality Data for Training Language Models." Proceedings of the 41st International Conference on Machine Learning (ICML 2024), Proceedings of machine learning research, vol. 235. https://arxiv.org/pdf/2402.09739. ↩

  7. Dodge, Jesse, Maarten Sap, Ana Marasović, et al. 2021. "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2021.emnlp-main.98. ↩

  8. Gururangan, Suchin, Dallas Card, Sarah K. Dreier, et al. 2022. "Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2022.emnlp-main.165. ↩