Reproducibility Versus Replicability

Two researchers can use the same words for different tests of reliability. In some disciplines, “replication” means rerunning the original analysis and “reproduction” means conducting a new study. In others, the labels are reversed or divided into several subtypes.

The solution is not to declare one vocabulary universal. It is to define your terms before you use them. This article adopts the 2019 National Academies convention because it draws a useful operational line: reproducibility concerns consistent computation with the same data, while replicability concerns consistent evidence from new data. That distinction prevents a common error: a result can be fully reproducible yet fail to replicate, and a conclusion can replicate even when the original computational record is unavailable.

Key points#

• Terminology varies, so every project should state its definitions.
• Reproducibility tests the path from the original inputs through the stated analysis to the reported output.
• Replicability tests the scientific claim with new observations.
• Neither identical numbers nor a repeated p value are sensible universal criteria.
• Design quality, measurement validity, uncertainty, and context still determine what the result means.

Reproducibility under the National Academies convention#

Reproducibility means obtaining consistent computational results using the same input data, methods, code, and analysis conditions; it asks: if a qualified analyst receives the materials that produced a table or figure, can that analyst obtain the reported values?

This is primarily a test of transparency, record keeping, and computational integrity: the required package can include raw or appropriately governed source data, a data dictionary, cleaning code, analysis code, software and package versions, configuration files, random seeds, and instructions. Complex work may require containers or another preserved computing environment.

“The code runs” is not enough. A pipeline may depend on an undocumented manual edit, a private lookup table, a local file path, or a software behavior that later changed. Reproducibility also fails when the output cannot be connected to the reported version of the analysis.

Version control helps establish that connection. A release, archived repository, or content hash can identify the code used for a paper. Automated tests can verify denominators, units, transformations, and expected outputs. A fresh analyst can then run the workflow without relying on the original analyst's memory.

Replicability under the same convention#

Replicability means obtaining consistent results across studies that gather new data to answer the same scientific question, and a replication may repeat the original methods closely or test the claim in a different but relevant setting.

The new data are the whole point. They break dependence on the original sample and can reveal chance findings, context-specific effects, measurement problems, and design sensitivities. A multicenter confirmation, a new experiment using the same protocol, or a prospectively registered study of the same hypothesis can all provide replication evidence.

Replicability does not require an identical effect estimate. Sampling variability alone makes exact equality unlikely. Populations, implementation, and measurement may also differ, so the replication should define what “consistent” means in advance, such as a compatible direction and magnitude with uncertainty that includes a clinically relevant range.

A second p value below 0.05 is a poor universal rule. A large, precise replication may find a smaller effect than the original and still support the core claim, while a small replication may produce the same point estimate with a wide interval and add little certainty. Effect sizes and confidence intervals are more informative.

One claim, four possible outcomes#

Consider a published analysis that reports an association between a biomarker and an outcome.

First, it may be reproducible and replicable. The code regenerates the original result, and a well-designed new study finds a compatible association. Confidence in the claim increases, subject to bias and relevance.

Second, it may be reproducible but not replicable. The original numbers are correct for the original data, yet a credible new study does not support the conclusion. Possible explanations include chance, bias, population differences, outcome definitions, or a truly context-dependent effect.

Third, it may be nonreproducible but replicable. The original computational record is incomplete or contains an error, while new studies repeatedly support a related conclusion, and the broader science may be sound even though the original report needs correction.

Fourth, it may be neither. The original output cannot be verified, and new evidence conflicts with it. Confidence should fall, but the appropriate response still depends on whether the problem is error, poor documentation, design weakness, or misconduct. Those four cells are why one label cannot stand in for the other.

Direct and conceptual replication#

A direct replication aims to preserve the original design and procedures as closely as practical. It tests whether the result reappears under similar conditions. Even direct work is never a perfect copy because time, participants, materials, personnel, and context change.

A conceptual replication tests the underlying hypothesis using different methods or operational definitions. Convergence across methods can show that a result is not an artifact of one assay or procedure. Disagreement can reveal that the supposed construct was defined too broadly.

Both are useful. Direct replication is stronger for diagnosing whether a specific procedure is reliable. Conceptual replication is stronger for testing the reach of a theory. Authors should state which goal they pursue and avoid redefining a conflicting result as “conceptual” after seeing it.

Reanalysis is not always reproduction#

An analyst may use the original data but intentionally change a method, correct an error, apply a different model, or ask a new question. That is a reanalysis. It can be scientifically valuable, but it should not be confused with regenerating the original result under the original specification.

A clear workflow preserves both. First, reproduce the original output if possible. Then run revised analyses and label the changes. This separates “the published number cannot be obtained” from “a different reasonable analysis gives a different answer.”

Robustness checks explore how conclusions respond to defensible analytic choices. They can reveal a fragile result, but a long collection of favorable alternatives can also hide selective reporting. The set of checks and the decision rule should be planned in advance and reported in full.

What a failure does and does not mean#

A failure to reproduce may reflect missing files, version drift, ambiguous instructions, data corruption, or a coding error. None automatically shows that someone acted dishonestly. Corrections and transparent documentation are normal mechanisms of science.

A failure to replicate may reflect low power, population differences, altered intervention fidelity, new measurement methods, or random variation. It may also show that the original effect was overestimated or absent. Investigators should compare designs and estimates before assigning a cause. Publication language often makes that harder by treating one study as a verdict, when a replication is another estimate rather than a courtroom appeal. The most informative synthesis considers the quality of all studies, their heterogeneity, and the plausibility of context-specific effects.

Materials that support reproducibility#

A strong archive contains:

• a protocol and dated analysis plan;
• data provenance and a machine-readable dictionary;
• code that rebuilds derived data from governed source data;
• dependency and software-version records;
• clear instructions and expected outputs;
• explanations for restricted data and a workable access process;
• an immutable release corresponding to the report.

Privacy, consent, security, community governance, and licensing can limit public data release. Reproducibility does not demand careless disclosure. Controlled-access repositories, secure environments, executable code with simulated data, and detailed metadata can allow scrutiny while respecting constraints.

Design features that support replicability#

Replication begins before data collection. The question, target population, outcomes, acceptable deviations, and consistency criteria should be defined prospectively. The sample should be large enough to estimate a meaningful effect with useful precision.

Teams should distinguish confirmatory from exploratory analyses, preserve null results, and document protocol deviations. Multisite work can examine variation rather than average it away. Measurement harmonization helps, but testing across more than one valid method can provide evidence of generality. In June 2026, NIH described an agency-wide initiative to elevate and incentivize replication and reproducibility. Infrastructure and funding can reduce practical barriers, but incentives must also value careful null findings, reusable software, data stewardship, and correction of the record.

Appraisal cautions#

Reproducible computation can reproduce bias perfectly. If the sample is unrepresentative, the outcome invalid, or the causal assumptions wrong, a pipeline you can rerun does not make the conclusion valid.

Replication also does not create certainty by vote count. Ten small studies with the same systematic bias do not outweigh one rigorous study merely because they are numerous. Appraisal needs design quality, effect estimates, uncertainty, and compatibility across contexts. And terminology itself should not become a gatekeeping exercise: when authors use the opposite convention but define it clearly, judge the operation they performed rather than policing the label.