Sample Size for Developing a Prediction Model

No fixed number of patients is enough to develop every prediction model, because the required sample depends on the outcome frequency, number of candidate predictor parameters, expected signal, model type, desired precision, missing data, and how the data will be used. The familiar rule of 10 events per variable compresses all of that into one ratio, and it can recommend too few observations in one setting and more than necessary in another.

Modern guidance starts with performance goals: the development sample should be large enough to keep overfitting small, keep optimism in apparent performance small, and estimate the population's overall outcome risk with useful precision. Continuous outcomes add goals for residual variation and the mean outcome. These criteria produce a study-specific calculation rather than a ceremonial rule.

Why small development samples overfit#

A model is fit to both signal and random variation in its development data. With many adjustable parameters and few informative outcomes, chance patterns receive coefficients. The model looks strong in the same data because it has learned their noise. Performance drops when the model meets new people.

That gap is optimism. In logistic regression it often appears as coefficients that are too large and predicted risks that are too extreme: high-risk estimates are too high, low-risk estimates are too low, and a calibration slope in new data falls below one. Flexible machine-learning methods can overfit as well, even when they do not report conventional coefficients.

Regularization helps by shrinking or constraining the fitted model, but it does not make sample size irrelevant; penalties must be tuned, and their performance is also uncertain with limited data. A model cannot learn stable subgroup patterns, rare combinations, or measurement variation that its dataset scarcely contains.

Events per variable is the wrong denominator#

The traditional shorthand divides the number of outcome events by the number of predictors. Several ambiguities follow. For a binary outcome, should the denominator include only the rarer outcome or both outcome groups? Does age plus age squared count as one variable or two parameters? Does a five-level categorical predictor count once or require four indicator coefficients?

The relevant count is candidate predictor parameters considered during modeling, and a continuous predictor modeled with a nonlinear spline might require several, a four-level category commonly requires three, and a prespecified interaction adds at least one. Data-driven screening does not erase parameters that were considered before selection, because the selection process itself can overfit.

Outcome frequency also changes the information available. Five thousand participants with a 1% outcome provide about 50 events. Five hundred participants with a 40% outcome provide 200. The first dataset is much larger but may carry less information about the event process.

Rules based on a fixed ratio ignore anticipated signal. A strong, well-established model with a few prespecified parameters can be estimated differently from a weak-signal search across many candidates; the ratio also ignores the precision needed for the intercept, which controls average predicted risk.

Three goals for binary and survival outcomes#

The framework proposed by Riley and colleagues for binary and time-to-event models uses at least three criteria.

First, target limited overfitting in predictor effects, often represented by an anticipated global shrinkage factor of at least 0.90; in plain language, the fitted effects should need no more than about 10% uniform shrinkage under the planning assumptions.

Second, target a small difference between apparent and optimism-adjusted model fit. For logistic models, one proposed criterion limits the difference in Nagelkerke R-squared to 0.05. This addresses apparent performance that looks materially better than expected new-data performance.

Third, estimate the overall outcome proportion or event rate precisely. A model can have acceptable relative effects yet a poorly estimated intercept if events are sparse. That error shifts every predicted probability.

The largest sample required by the applicable criteria becomes the planning minimum. This matters because different settings are driven by different constraints. A rare outcome may make intercept precision decisive. A large candidate set with modest expected signal may make shrinkage decisive.

What must be specified before calculation#

Before you can calculate anything for a binary outcome you need the anticipated outcome proportion, the number of candidate predictor parameters, and the expected model fit, commonly expressed through Cox-Snell R-squared. A survival calculation also needs the event rate, time horizon, and follow-up information. A continuous-outcome calculation uses expected R-squared and assumptions about outcome variation.

These are planning inputs, not facts. Ground them in systematic reviews, prior models, pilot data, or conservative assumptions. Run scenarios when uncertainty is substantial. If one plausible event rate requires twice the sample of another, that range belongs in the protocol.

Expected R-squared is often unavailable. Guidance offers conservative approximations based on the maximum possible value for the outcome proportion, but the assumption should be visible, and choosing an optimistic value solely to shrink the required sample defeats the method.

Software such as the pmsampsize package implements these calculations for common regression settings. A printed result is not a substitute for understanding the inputs. The report should give you enough detail to reproduce the number yourself.

A worked lesson from published guidance#

The BMJ tutorial gives an illustrative binary model with a 5% outcome, 30 candidate parameters, and a conservatively chosen Cox-Snell R-squared of 0.05. Under those inputs, the sample needed to satisfy the proposed criteria is 5,249 participants with about 263 events, equivalent to 8.75 events per parameter.

The example is useful because 8.75 is below the old rule of 10, yet it comes from explicit performance targets. In a time-to-event example in the same tutorial, the required ratio is 23.1 events per parameter, well above 10. The method does not try to prove one ratio correct. It shows why no one ratio can govern both settings.

These numbers are not portable defaults. Change the outcome frequency, anticipated fit, number of parameters, or precision target and the sample you need changes with them. The reproducible part is the reasoning, not the answer.

Missing data reduce usable information#

A plan based on 2,000 participants may fail if 25% lack a key predictor or follow-up, and complete-case analysis can reduce effective sample size and distort the population when missingness is related to outcomes or predictors. Multiple imputation retains information under explicit assumptions, but it does not replace measurements that were never collected.

Inflate recruitment for expected missingness and loss to follow-up. More importantly, design data collection to prevent avoidable missingness. Record why values are absent, include the outcome and relevant predictors in imputation models, and carry the full modeling process through resampling.

Rare categories can create local scarcity even when the overall sample clears a formula, so a model intended for multiple sites, devices, or sociodemographic groups needs enough information to estimate relevant variation. Fairness cannot be repaired by reporting unstable subgroup metrics after development.

Splitting can make a modest dataset smaller twice#

A random development-test split is familiar, but it withholds information from model fitting and often leaves too few outcomes for precise evaluation. A 70:30 split turns 100 events into about 70 for development and 30 for testing. Neither side may answer its question well.

For internal validation, bootstrap resampling or carefully performed cross-validation can use all observations for development while estimating optimism. Every modeling action, including imputation, preprocessing, feature selection, and tuning, must occur within each resample. Otherwise information leaks from validation data into training.

Internal validation estimates reproducibility under the development-data process. It does not establish transport to a new hospital, time period, country, device, or workflow. External evaluation needs a separate dataset and its own sample-size calculation based on the precision of calibration, discrimination, and threshold measures. Large datasets can support internal-external cross-validation, where sites or clusters are held out in turn, and that approach examines heterogeneity and transport, but it still requires enough clusters and events within held-out groups.

Machine learning does not remove the planning problem#

The Riley formulas were developed for specified regression settings. Highly adaptive methods may require simulation-based planning or learning-curve analysis because effective complexity is harder to count. The core questions remain: what outcome information is available, how many choices can the algorithm make, how stable are predictions, and how precise must evaluation be?

Calling a dataset “big data” gives no answer. Millions of rows from a few hundred patients may have a patient-level sample of only a few hundred. Repeated images and time points are correlated. Splitting them across training and test sets can leak patient identity and exaggerate performance.

Effective sample size also depends on label quality. A large dataset of noisy billing codes may provide less clinically relevant information than a smaller cohort with careful outcome adjudication, and sample size, representativeness, and measurement validity are separate dimensions.

How to read a model-development paper#

TRIPOD+AI asks authors to explain how study size was reached, justify sufficiency, report participant and event counts for each analysis, and describe data partitioning. You can turn those requests into an audit:

1. What population, outcome, and horizon was modeled?
2. How many participants, events, and nonevents entered each stage?
3. How many candidate parameters were considered before selection?
4. What assumptions drove the sample-size calculation?
5. How were missing data, nonlinear terms, interactions, and clustering handled?
6. Did internal validation repeat every modeling step?
7. Was performance estimated with confidence intervals and calibration plots?
8. Was external evaluation separate and adequately sized?

A paper that tells you “more than 10 events per variable” has given you a ratio, not a justification. A credible report connects sample size to expected model signal, overfitting control, and precision.

The guides to reading a calibration plot and appraising a clinical prediction model cover the next steps. The site's research overview explains the wider evidence-methods program.

References#

1. Minimum sample size for binary and time-to-event prediction models
2. Minimum sample size for continuous-outcome prediction models
3. BMJ tutorial for prediction-model sample size
4. TRIPOD+AI reporting statement
5. Systematic review of sample-size practice in binary prediction models