Skip to content

Software Quality

Annotation agreement

Measuring whether the people labeling a dataset agree with each other and with known answers, before the labels are trusted.

Annotation agreement measures whether the people labeling a dataset produce the same labels for the same items, before the labels train or evaluate a model. Several annotators label a sample of items independently, and a coefficient summarizes how often they agree beyond what chance would produce. Low agreement means the label depends on who assigned it: the guidelines are ambiguous, the annotators are untrained or inattentive, or the items have no single right answer. The oracle is the other annotators, not a model, which separates it from label-error detection, where a model trained on the data disputes individual labels after labeling is done.

Kinds of check

  • Agreement coefficient {#agreement-coefficient}. Percent agreement counts matching labels; Cohen's kappa (two annotators), Fleiss' kappa (several) and Krippendorff's alpha (any number of annotators, missing labels, ordered or numeric scales) subtract the agreement chance alone would give. The thresholds in common use come from content analysis, where a value above 0.8 counts as good reliability and one between 0.67 and 0.8 allows tentative conclusions (Artstein and Poesio 2008)1.
  • Gold questions {#gold-questions}. Items with a known answer are mixed into the queue without being marked, and an annotator whose accuracy on them falls below a bar is removed, retrained or given less weight.

What a team does with disagreement is a separate step: majority vote over repeated labels, a model that weights each annotator by estimated accuracy, or adjudication, in which experts discuss each disputed item until they agree.

What it catches

  • Ambiguous guidelines. Categories annotators read differently, visible as agreement that is low on some classes and high on others.
  • Unreliable annotators. Annotators who guess, rush or misunderstand the task, visible on gold questions and in agreement with the others.
  • Items with no single answer. Items where informed people disagree, whose majority label is a vote rather than a fact.

What it does not catch:

  • a mistake every annotator makes, which agrees perfectly;
  • a wrong label on an item labeled only once, which label-error detection looks for.

Tools

  • krippendorff — Python package computing Krippendorff's alpha for any level of measurement, with missing labels.
  • statsmodels — Python statistics library; Cohen's kappa, weighted kappa and Fleiss' kappa.
  • NLTK — Python language-processing library whose agreement module computes kappa, multi-annotator kappa, pi and alpha.
  • irr and irrCAC — R packages for kappa, Fleiss' kappa, alpha and the intraclass correlation; irrCAC adds Gwet's coefficients and their variances.
  • Crowd-Kit — Python library aggregating repeated labels (majority vote, Dawid–Skene and other models) and computing agreement.
  • cleanlab — its multi-annotator module derives a consensus label, a quality score for each annotator and the agreement on each item.

When to use, when not

Use:

  • On a pilot batch before labeling at scale, while the guidelines can still change.
  • On every labeled dataset that evaluates a model, since a disputed test label scores a model against a coin toss among annotators.
  • With gold questions in any crowdsourced task, where some annotators will not read the instructions.

Don't:

  • As evidence that labels are correct, since agreement measures reproducibility, and annotators trained on the same flawed guideline agree on its errors.
  • To discard items people genuinely disagree on: their disagreement is a property of the item, and a model scored only on unanimous items looks better than it is.

Evidence

  • Repeated labels help only above chance. With simulated labelers, a majority vote was more accurate than one label only when each labeler was right more than half the time, and below that more labelers made it worse; the gain also shrank with each labeler added (Sheng et al. 2008)2.
  • Three votes halved the error of one. Each CIFAR-10 training image was labeled by three crowd workers: a single worker's labels were 17.2% to 18.1% wrong, the majority label 9.03% (Wei et al. 2022)3.
  • Non-expert crowds agreed with experts. On five language tasks labeled on Mechanical Turk, non-expert annotations agreed highly with expert gold labels (Snow et al. 2008)4.
  • Adjudication found what the majority missed. Three retina specialists grading diabetic retinopathy each had a sensitivity of 74.4% to 82.1% against the adjudicated grade; their majority vote reached 88.1%, leaving 11.9% of moderate-or-worse cases that only adjudication detected (Krause et al. 2018)5.
  • Majority labels hide disagreement. With 100 annotations per example, the original majority label no longer matched the prevailing human opinion for 10% to 31% of the language-inference examples collected; two of the three sets were selected for low original agreement. Models were near perfect where people agreed and barely better than chance where they did not (Nie et al. 2020)6.
  • Disagreement lowered a reported score from .95 to .73. On a comment toxicity task where models report .95 ROC AUC, even a classifier predicting every aggregated label perfectly would score .73 once annotator disagreement is counted (Gordon et al. 2021)7.

Classification

  • Data-quality characteristics: Consistency (labels that annotators cannot reproduce contradict each other, and a model trained or scored on them learns who labeled), Accuracy (an annotator who fails questions with known answers gives wrong labels on the rest).
  • Area: Datasets labeled by people — crowdsourced and expert annotation for training and evaluating models, and the reference grades that clinical and scientific measurements are scored against.
  • Guarantee: Heuristic, Empirical (Gold questions).

Referenced by

References

  1. Artstein, Ron, and Massimo Poesio. 2008. "Inter-Coder Agreement for Computational Linguistics." Computational Linguistics 34 (4): 555–96. https://doi.org/10.1162/coli.07-034-R2. ↩

  2. Sheng, Victor S., Foster Provost, and Panagiotis G. Ipeirotis. 2008. "Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers." Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. https://doi.org/10.1145/1401890.1401965. ↩

  3. Wei, Jiaheng, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. 2022. "Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations." International Conference on Learning Representations (ICLR 2022). https://arxiv.org/pdf/2110.12088. ↩

  4. Snow, Rion, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. "Cheap and Fast — but Is It Good? Evaluating Non-Expert Annotations for Natural Language Tasks." Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/D08-1027.pdf. ↩

  5. Krause, Jonathan, Varun Gulshan, Ehsan Rahimy, et al. 2018. "Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy." Ophthalmology 125 (8): 1264–72. https://doi.org/10.1016/j.ophtha.2018.01.034. ↩

  6. Nie, Yixin, Xiang Zhou, and Mohit Bansal. 2020. "What Can We Learn from Collective Human Opinions on Natural Language Inference Data?" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10.18653/v1/2020.emnlp-main.734. ↩

  7. Gordon, Mitchell L., Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021. "The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics in Line with Reality." Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445423. ↩