# Statistical integrity tests A statistical integrity test asks whether a dataset as a whole looks like genuine measurement. Fabricated, manipulated or clipped data can hold only plausible values and still carry a signature in its distribution: groups that should differ only by chance differ too much or too little, values pile up at a limit, or final digits appear in proportions no instrument produces. The test computes a statistic over the whole dataset and compares it with what genuine data would give, so a flag is an improbability to investigate rather than a verdict. It judges whether the data can be believed and whether it has the resolution it states, which [ISO/IEC 25012](https://quality.stereobooster.com/characteristics.md) calls credibility and precision, rather than whether any one value is correct. ## Kinds of test - **Balance between randomized groups.** In a randomized trial, the groups differ at baseline only by chance, so baseline comparisons between them should produce neither too many extreme differences nor too few. - **Variance and correlation.** Invented values tend to reproduce a variable's mean more easily than its spread and its relations with other variables. - **Digits.** Leading digits of many natural quantities follow Benford's law, and final digits of precise measurements are close to uniform. People recording values by hand round to preferred digits, so digit preference alone is not evidence of misconduct; it should, however, look the same in groups formed by randomization (Al-Marzouki et al. 2005)[^almarzouki2005]. - **Density at a bound.** A test for a jump in a variable's density at a known cutoff detects values pushed across a threshold, such as scores nudged to just qualify (McCrary 2008)[^mccrary2008]. Values clipped to a limit by a pipeline, or held at a sensor's maximum, pile up at the bound in the same way. ## Digit preference and heaping Genuine data can be recorded more coarsely than it claims. People reading an instrument round to a preferred final digit, most often 0; respondents give their age as a multiple of five; records are georeferenced to a grid or to a few decimals. The values are believable but carry less resolution than the field states. The digit tests detect it: final digits or decimals are counted and compared with the near-uniform spread that measurement at the stated resolution produces, and heaping indices such as Whipple's measure the excess of reported ages at multiples of five. Rounding cannot be undone once recorded, so the check does most good while data is still being collected, where its result can go back to the people recording the values. ## What it catches - **Fabricated records.** Values invented rather than measured, in a trial, a survey or a submitted dataset. - **Subverted allocation.** Groups described as randomized that were formed some other way. - **Values pushed across a threshold.** Measurements adjusted to clear an eligibility cutoff or a reporting limit. - **Coarse recording.** Readings rounded to a preferred digit, ages heaped at multiples of five, and coordinates rounded to a grid. - **Clipped or saturated values.** A column capped at a limit by a transformation, or a sensor reading held at its maximum. What it does **not** catch: - a fabrication that reproduces genuine data's variance, digit frequencies and correlations; - a single wrong value, which [data error detection](https://quality.stereobooster.com/data-error-detection.md) looks for; - an error that leaves the distribution unchanged, such as a constant offset in every value. ## Checking reported statistics Two related checks test the numbers a research paper reports rather than a dataset, so they serve reviewers of papers rather than teams that hold data. GRIM tests whether a reported mean is possible for integer-valued data from the stated number of participants; around half of the 71 psychology articles it could test appeared to contain at least one inconsistent mean (Brown and Heathers 2017)[^brown2017]. statcheck recomputes each p-value from the test statistic and degrees of freedom a paper reports and flags those that do not match; half of the psychology papers using significance tests from 1985 to 2013 contained at least one inconsistent p-value, and one in eight a gross inconsistency that may have changed the conclusion (Nuijten et al. 2016)[^nuijten2016]. ## Tools - **[rddensity](https://github.com/rdpackages/rddensity)** — R, Stata and Python package that tests for a discontinuity in a variable's density at a cutoff. - **[pvanalytics](https://github.com/pvlib/pvanalytics)** — Python quality-assurance library for solar power data; labels clipping in power output series from plateau levels or a power threshold. - **[CoordinateCleaner](https://github.com/ropensci/CoordinateCleaner)** — R package for cleaning species occurrence records; its `cd_round` test flags datasets whose coordinates show decimal periodicity or an excess of zero decimals. - **[DemoTools](https://github.com/timriffe/DemoTools)** — R package for demographic data; computes age-heaping indices such as Whipple's and Myers'. ## When to use, when not **Use:** - Before relying on data someone else collected, such as a submitted trial, a vendor feed or another lab's dataset. - On the raw records where they are available, since variance, correlation and digit tests need individual values. - On a variable with a known limit or threshold, where clipping or manipulation leaves values piled at that value. **Don't:** - As a verdict on its own. An improbable pattern can come from error, stratified allocation or correlated variables, and a flag starts an investigation. - Benford's law on data with no reason to follow it. Election returns, for example, conform to it or deviate from it regardless of fraud. - A test of final digits against uniformity on data recorded by hand, where rounding to preferred digits is normal. Compare the groups with each other instead. ## Evidence - **Baseline balance flagged retracted trials far more often than others.** Across 5,087 randomized trials in eight journals, 15.2% of unretracted trials had a baseline p-value within 0.05 of 0 or 1, against 10% expected, and 43% of retracted trials did. A threshold of 1 in 10,000 flagged 11% of retracted trials and 1.6% of unretracted ones (Carlisle 2017)[^carlisle2017]. - **Raw records exposed false data far more often than summary tables.** Of 526 trials submitted to one journal, false data were found in 67 of the 153 that supplied individual patient data (44%) and in 6 of the 373 that did not (2%) (Carlisle 2021)[^carlisle2021]. - **Group differences and digits exposed one fabricated trial.** In a dietary trial, variances differed between the randomized groups for 16 of 22 baseline variables and means for 10, and final-digit distributions differed for 14 of 22. A drug trial analyzed the same way showed none of these differences, and the authors judged that no explanation other than fabrication was likely (Al-Marzouki et al. 2005)[^almarzouki2005]. - **Raw-data tests separate fabricated from genuine data, except digit tests.** With 21 genuine Stroop-task datasets and 28 fabricated by researchers for the same effect, variance analyses of the raw data performed excellently, multivariate associations fairly to excellently, and digit analyses at chance level. The study is an unreviewed preprint (Hartgerink et al. 2019)[^hartgerink2019]. - **Benford's law does not detect election fraud.** On simulated elections and on elections whose fraud status was known from other investigations, conformity with and deviation from Benford's law followed no pattern, and its success rate either way was essentially that of a coin toss (Deckert et al. 2011)[^deckert2011]. - **Fabricated numbers passed a first-digit test.** Students asked to fabricate regression coefficients produced first digits that declined as Benford's law predicts, while their second digits deviated significantly from it (Diekmann 2007)[^diekmann2007]. - **Office blood pressure readings end in zero far more often than chance.** From 2015 to 2019, the share of US office visits for treated hypertension with a systolic reading ending in 0 fell from 41.7% to 37.7%, where 10%–20% would be expected without bias (Foti et al. 2021)[^foti2021]. - **Feeding digit preference back to trial centers reduced it.** In a multinational trial whose monitoring committee sent each center its digit-preference charts every year, the share of first sitting systolic readings ending in 0 fell from 42.4% in the year before a center's first randomization to 22% in its sixth year (Wingfield et al. 2002)[^wingfield2002]. - **A rounding test flagged nearly a fifth of biodiversity datasets.** CoordinateCleaner's rounding test flagged 18.5% of 966 GBIF datasets with more than 100 occurrence records as possibly biased by rasterized or rounded coordinates (Zizka et al. 2019)[^zizka2019]. ## Classification - **Data-quality characteristics:** Credibility (a fabricated, manipulated or clipped dataset cannot be believed, though each value looks plausible), Precision (values rounded to a preferred digit or a grid carry less resolution than the instrument or field records), Accuracy (clipped and fabricated values do not represent what was measured). - **Area:** Datasets a team did not collect itself and must decide whether to trust — submitted trial data, a vendor feed, data shared by another lab — measured variables with a known limit or threshold, such as a sensor's range or an eligibility cutoff, and measurements recorded by hand or rounded on entry, such as blood pressure readings, ages and coordinates. - **Guarantee:** Empirical. ## Referenced by - [Data quality](https://quality.stereobooster.com/data-quality.md) · Data quality ## References [^almarzouki2005]: Al-Marzouki, Sanaa, Stephen Evans, Tom Marshall, and Ian Roberts. 2005. "[Are These Data Real? Statistical Methods for the Detection of Data Fabrication in Clinical Trials](https://researchonline.lshtm.ac.uk/id/eprint/13500/1/267.full.pdf)." *BMJ* 331 (7511): 267–70. . [^mccrary2008]: McCrary, Justin. 2008. "[Manipulation of the Running Variable in the Regression Discontinuity Design: A Density Test](http://www.nber.org/papers/t0334.pdf)." *Journal of Econometrics* 142 (2): 698–714. . [^brown2017]: Brown, Nicholas J. L., and James A. J. Heathers. 2017. "[The GRIM Test: A Simple Technique Detects Numerous Anomalies in the Reporting of Results in Psychology](https://doi.org/10.1177/1948550616673876)." *Social Psychological and Personality Science* 8 (4): 363–69. . [^nuijten2016]: Nuijten, Michèle B., Chris H. J. Hartgerink, Marcel A. L. M. van Assen, Sacha Epskamp, and Jelte M. Wicherts. 2016. "[The Prevalence of Statistical Reporting Errors in Psychology (1985–2013)](https://pure.uva.nl/ws/files/21277070/The_prevalence_of_statistical_reporting_errors_in_psychology.pdf)." *Behavior Research Methods* 48 (4): 1205–26. . [^carlisle2017]: Carlisle, J. B. 2017. "[Data Fabrication and Other Reasons for Non-Random Sampling in 5087 Randomised, Controlled Trials in Anaesthetic and General Medical Journals](https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/anae.13938)." *Anaesthesia* 72 (8): 944–52. . [^carlisle2021]: Carlisle, J. B. 2021. "[False Individual Patient Data and Zombie Randomised Controlled Trials Submitted to Anaesthesia](https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/anae.15263)." *Anaesthesia* 76 (4): 472–79. . [^hartgerink2019]: Hartgerink, Chris H. J., Jan G. Voelkel, Jelte M. Wicherts, and Marcel A. L. M. van Assen. 2019. *[Detection of Data Fabrication Using Statistical Tools](https://osf.io/jkws4/download)*. PsyArXiv preprint. . [^deckert2011]: Deckert, Joseph, Mikhail Myagkov, and Peter C. Ordeshook. 2011. "[Benford's Law and the Detection of Election Fraud](https://www.cambridge.org/core/services/aop-cambridge-core/content/view/3B1D64E822371C461AF3C61CE91AAF6D/S1047198700012808a.pdf/div-class-title-benford-s-law-and-the-detection-of-election-fraud-div.pdf)." *Political Analysis* 19 (3): 245–68. . [^diekmann2007]: Diekmann, Andreas. 2007. "[Not the First Digit! Using Benford's Law to Detect Fraudulent Scientific Data](https://doi.org/10.1080/02664760601004940)." *Journal of Applied Statistics* 34 (3): 321–29. . [^foti2021]: Foti, Kathryn E., Lawrence J. Appel, Kunihiro Matsushita, Josef Coresh, G. Caleb Alexander, and Elizabeth Selvin. 2021. "[Digit Preference in Office Blood Pressure Measurements, United States 2015–2019](https://doi.org/10.1093/ajh/hpaa196)." *American Journal of Hypertension* 34 (5). . [^wingfield2002]: Wingfield, David, Jonathan Cooke, Lut Thijs, et al. 2002. "[Terminal Digit Preference and Single-Number Preference in the Syst-Eur Trial: Influence of Quality Control](https://doi.org/10.1097/00126097-200206000-00005)." *Blood Pressure Monitoring* 7 (3). . [^zizka2019]: Zizka, Alexander, Daniele Silvestro, Tobias Andermann, et al. 2019. "[CoordinateCleaner: Standardized Cleaning of Occurrence Records from Biological Collection Databases](https://www.research-collection.ethz.ch/server/api/core/bitstreams/70806f13-c9cb-4ba7-94e8-d70aa6f38a4d/content)." *Methods in Ecology and Evolution* 10 (5). . ## Acronyms - GBIF — Global Biodiversity Information Facility - GRIM — granularity-related inconsistency of means