What statcheck and GRIM Can Reveal From a Paper's Reported Numbers

A published paper can contradict itself even when its raw data are unavailable. statcheck compares a reported test statistic and degrees of freedom with the reported p value. GRIM checks whether a rounded mean could be produced by the stated number of integer observations, and a mismatch is a reason to inspect the report, not proof of how the mismatch arose.

Key points#

• Internal-consistency checks ask whether numbers printed in the same paper can all be true at once.
• statcheck works only for statistical results in formats it recognizes and can misread corrected or one-sided tests.
• GRIM applies to means based on integer-valued observations with known sample size and adequate reporting precision.
• Typos, rounding, adjusted analyses, changing denominators, and extraction errors can all create flags.
• The responsible next step is verification and correction, not an accusation based on an automated screen.

The logic of checking without a dataset#

Many summary statistics are mathematically linked: if a paper reports a t statistic, its degrees of freedom, and a p value, the first two determine the third under a specified test. If it reports the mean of 10 integer scores, that mean must come from a total divided by 10. Only certain rounded values are possible.

These constraints make a report partially self-auditing, which is useful when you have the paper and nothing else, though the checks cannot determine whether the underlying observations were collected properly or whether the chosen model was appropriate. They can show where the published account needs explaining.

How statcheck works#

statcheck searches text for standardized results such as t, F, chi-squared, z, correlation, and some meta-analytic Q tests; it extracts the test statistic, degrees of freedom, and p value, recalculates the p value, and compares the result with what the paper reports.

For example, suppose a paper prints t(38) = 2.10, p = .042. A calculator can derive the two-sided p value from t equals 2.10 with 38 degrees of freedom. If the recomputed value rounds differently enough to conflict with .042, statcheck flags the line. A “decision inconsistency” is more consequential: the reported and recalculated values fall on opposite sides of a threshold such as .05.

The software is efficient because it does not interpret the entire analysis. That is also its limitation. It recognizes particular reporting patterns, so tables, unusual notation, images, malformed PDF text, and statistics with extra parameters may all slip past it, so a paper that gives you no flags has not received a complete statistical audit.

Why a statcheck flag may be legitimate#

The reported p value may have been adjusted while the displayed statistic was not. Multiple-comparison corrections, one-sided tests, Greenhouse-Geisser or Huynh-Feldt corrections, and other procedures can break the simple relationship the software assumes. Authors sometimes report corrected degrees of freedom but an uncorrected statistic, or state the correction elsewhere in the methods.

Rounding can also matter near a threshold. A statistic printed to two decimals contains less information than the number used by the software that generated the original p value. Extracting text from a PDF can turn a minus sign, superscript, or decimal into something else. The paper may switch sample size across analyses because of missing observations.

These possibilities do not make the flag useless. They tell you what to check. Clear reporting of adjusted degrees of freedom, exact p values where appropriate, test direction, and correction method reduces false alarms and makes the analysis reproducible.

How GRIM works#

GRIM stands for granularity-related inconsistency of means. If observations are integers, their sum is an integer. For a sample of 20 people answering one integer-scored item, the unrounded mean must increase in steps of 1 divided by 20, or 0.05. A reported mean of 3.42 cannot come from that setup when rounded to two decimal places because the possible values include 3.40 and 3.45, not 3.42.

The arithmetic changes when a mean combines several integer items. If 20 participants each complete 5 items and all 100 item scores are averaged directly, the granularity can be 1 divided by 100. If each participant first receives a scale score and those scores are averaged, missing items or weighting may alter the denominator. The paper's scoring rule matters.

GRIM therefore requires several facts:

• the data feeding the mean were integer-valued,
• the relevant sample size is known,
• the number of items and scoring method are understood,
• weights or transformations did not create fractional observations,
• the reported precision is sufficient for the test.

Continuous measurements, adjusted means, model-based estimates, imputed values, weighted survey data, and many composite scores fall outside the simple check.

A flag is not a finding of misconduct#

An impossible or mismatched number establishes a reporting problem only after the calculation and context are verified. It does not identify intent. Plausible explanations include transcription, a swapped table label, a denominator changed after missing-data handling, a correction not described, or a mismatch between an early and final analysis file.

More serious explanations are also possible, but jumping directly to them is both unfair and methodologically weak. Research integrity processes need evidence, an opportunity for authors to respond, and review of underlying records where warranted. The correct language is proportionate: “the reported mean is inconsistent with the stated sample size under this scoring assumption,” not “the data were fabricated.”

A verification workflow for readers and reviewers#

When a tool raises a flag:

1. Return to the exact sentence or table in the source file.
2. Confirm that text extraction preserved symbols, decimals, and degrees of freedom.
3. Identify whether the test was one-sided, adjusted, or corrected.
4. Recompute the result using the displayed values and the stated method.
5. Check footnotes for changing sample sizes, missing data, or alternative denominators.
6. For GRIM, reconstruct the scale and rounding rule.
7. Decide whether the issue changes a significance label, effect estimate, or conclusion.
8. Seek clarification through ordinary editorial channels.

If the error is real, its importance depends on consequence. A p value mistyped as .034 instead of .043 may not change the interpretation. A value printed as .004 instead of .40 could. A correction notice may be sufficient for a local error, while a pattern affecting central findings deserves broader review.

Using automated checks before publication#

The most constructive use is preventive. Authors can run consistency checks before submission, journals can integrate them into technical screening, and analysts can generate narrative results directly from validated tables rather than retyping numbers.

A 2024 quasi-experimental study covering more than 7,000 articles and 147,000 extracted statistics reported a steeper decline in inconsistencies at journals that incorporated statcheck into peer review than at matched comparison journals. Because the design was not randomized, it cannot prove that the tool alone caused the change. It nevertheless supports screening as one component of a broader quality process.

Automation should complement, not replace, statistical review. statcheck cannot judge whether the outcome was switched, the model assumptions were credible, missing data were handled appropriately, or the effect mattered clinically. GRIM cannot validate continuous data or prove that a reported sample size applies to the mean. Both tools see arithmetic relations, and you still have to supply the scientific question.