When More Health Data Is Not Better Evidence

Health systems now hold millions of encounters, claims, and images. They hold waveforms, prescriptions, messages, and device readings. Scale can uncover rare safety signals, support narrower subgroup estimates, and train complex models. But volume is not the same as validity.

More observations reduce random sampling noise when the observations represent the target and are measured appropriately. They do not automatically repair a mislabeled outcome, an unfair proxy, or a selected population. They do not repair a changing workflow or a device that measures some people less accurately.

A very large biased dataset can produce narrow confidence intervals around the wrong answer. The central question is not “How many rows?” It is “Are these data fit for this purpose?”

Precision and validity answer different questions#

Precision concerns how much an estimate would vary across repeated samples under the model. With more appropriately sampled observations, standard errors often shrink. Validity concerns whether the estimate captures the intended quantity without important systematic error.

Suppose a blood-pressure cuff reads 8 millimeters of mercury too high for every person. Measuring one million people will estimate the biased average very precisely. The large sample does not calibrate the cuff.

The same principle applies to algorithms. A model can achieve a stable area under the curve while predicting a proxy that does not represent the actual care need. It can be reproducibly wrong because the data-generating process is reproducibly biased.

Begin with the question and estimand#

Before you acquire more data, define the target population, the treatment or predictor, and the comparator. Define the outcome, the time horizon, and the summary measure. In causal research, this target is often called the estimand. In prediction, define the exact future event and moment when the prediction will be used.

“Predict hospital risk” is underspecified. Does risk mean admission in 30 days, preventable admission, death, cost, length of stay, or need for outreach? Is the model used before triage, after laboratory results, or at discharge? Data available after that moment cannot enter a real-time predictor without leakage. The relevant sample follows from the question. Ten thousand deeply characterized patients may answer it better than ten million billing records missing the outcome and key confounders.

Electronic records record care, not biology alone#

An electronic health record is created to deliver and document care. A diagnosis code can reflect suspicion, rule-out work, billing, problem-list carry-forward, or confirmed disease. A medicine order does not prove it was dispensed or taken. A missing laboratory value may mean the test was unnecessary, inaccessible, performed elsewhere, or omitted during unstable care.

Claims are optimized for reimbursement; they can capture billed encounters across a payer network but may lack clinical detail, change when insurance changes, and use codes shaped by payment rules. Death, symptom severity, over-the-counter use, and social context may be incomplete.

FDA's guidance on EHR and claims data treats relevance and reliability as question-specific. It asks whether key elements are available, complete, accurately defined, temporally appropriate, and traceable. The guidance contains nonbinding recommendations; it does not declare every large database suitable for regulatory evidence.

The target can be wrong even when prediction is strong#

A commercial population-health algorithm studied by Obermeyer and colleagues predicted future healthcare cost as a proxy for health need. The Science study found that, at the same risk score, Black patients were substantially sicker than White patients.

The problem was not merely too few Black records. Spending was lower for Black patients at a given level of illness because access and care patterns were unequal. Cost therefore encoded a social and health-system process. Replacing the target with health indicators greatly reduced the disparity in simulated program allocation. That is a failure of target definition: more years of the same cost records could strengthen the proxy relation without making cost equal to need.

Measurement can differ across groups#

Large datasets often treat device outputs as objective truth. Yet device error can depend on physiology, skin pigmentation, and movement. It can depend on perfusion, placement, software, and calibration.

A 2020 NEJM study of pulse oximetry compared pulse-oximeter readings with arterial measurements. Among readings of 92 to 96 percent, occult hypoxemia was detected more often in Black patients than White patients in two cohorts, and the finding showed differential measurement error in a widely used signal.

Training on millions of such readings without recognizing the error can teach a model that a biased measurement is truth. Better data may require a reference standard, device metadata, or skin-pigmentation measurement. It may require paired samples or redesigned hardware, not just additional rows.

Selection bias can grow with scale#

A hospital database includes people who reached that hospital and generated a record. It excludes those who could not access care, used another system, or were not tested. A biobank includes people who consented and remained reachable. A wearable dataset includes users who bought, wore, synced, and charged the device.

If inclusion relates to both predictors and outcomes, associations within the dataset can differ from those in the target population. Conditioning on a healthcare event can also create collider bias, making two otherwise separate causes appear related. A million selected participants remain selected. The remedy may be a different sampling frame, weighting based on defensible assumptions, or linkage. It may be active follow-up, or a narrower claim about whom the evidence represents.

Missingness is information#

Missing values are rarely random in health data. A test is ordered because a clinician suspects something. A patient misses follow-up because of illness, work, transport, cost, or improvement. A vital sign is absent because a device failed or the clinical situation changed.

Simple imputation can erase these mechanisms. Adding a missingness indicator may improve prediction while exploiting a local workflow that fails elsewhere. Complete-case analysis can select an unrepresentative subset.

Plot the missingness by site, by time, by outcome, and by patient group. Document why each variable exists at all, and state which missing-data mechanism you think is plausible. A sensitivity analysis is usually more honest than one filled-in dataset.

Duplicates and dependence create false sample size#

One patient can contribute hundreds of laboratory results, images, notes, and admissions. Treating them as independent observations understates uncertainty, and a common split error puts records from the same person in both training and test sets, allowing the model to recognize patient-specific patterns.

Near-duplicate images, copied notes, multiple crops of the same scan, and repeated specimens can produce similar leakage. Multi-site datasets may include overlapping referral populations or transferred images.

Split on whatever you intend to generalize to. If the future task concerns new patients, split by patient. If it concerns future care, use a temporal test. If it concerns a new hospital, hold out sites. Clustered analysis should account for repeated observations.

Label leakage makes performance look extraordinary#

Leakage occurs when a predictor contains information that would not be available at the intended prediction time or directly reveals the outcome, and a discharge diagnosis, treatment started after deterioration, or note written after a test result can make retrospective prediction easy.

Subtle leakage can come from ordering patterns, timestamps, or care location. It can come from filename, scanner annotation, or a code added during outcome adjudication. A model may detect that clinicians already acted rather than forecast the clinical event.

Draw a timeline for every variable and freeze the prediction moment, then reconstruct what the user would genuinely know at that instant and nothing after it. Prospective silent testing often shows you gaps that a random retrospective split conceals.

Big features invite big overfitting#

When researchers test thousands of variables, some correlate with the outcome by chance. Flexible models can fit noise, particularly when the number of independent outcome events is much smaller than the row count.

Regularization, nested cross-validation, held-out test sets, prespecified analyses, and correction for multiple testing help. They do not replace external validation. Hyperparameter choices made after repeatedly checking a test set effectively train on that test set. Report the full search process, not only the winning model. A modest result from a locked external evaluation is more informative than the best of hundreds of unreported experiments.

Drift makes yesterday's abundance stale#

Populations, viruses, and treatment change. So do coding, devices, referral pathways, and clinician behavior. A relationship learned before a new drug, laboratory method, or clinical guideline may weaken after adoption.

Research on dataset shift in machine-learning biomarkers shows how performance can break when the joint distribution changes. Shift detection, recalibration, external testing, and ongoing outcome monitoring are necessary. But not every shift can be fixed mathematically. A huge historical dataset can overweight obsolete practice. Sometimes a smaller recent sample, deliberately combined with historical data, better reflects the deployment setting.

Google Flu as a scale warning#

Google Flu Trends used enormous search-query data to estimate influenza-like illness. The 2014 Parable of Google Flu described substantial overestimation and highlighted big-data hubris, algorithm dynamics, and instability in the relation between searches and illness.

Search behavior changed with media attention and platform design. The underlying search algorithm also evolved. The proxy was produced by a system that influenced what people searched, rather than a fixed sensor of disease. The lesson is not that digital traces are useless. It is that they need transparent methods, stable reference surveillance, recalibration, and combination with domain knowledge.

More data can increase governance risk#

Collecting additional variables increases privacy, security, consent, linkage, and re-identification concerns. A field that offers little analytic value can still cause harm if disclosed. Long retention and broad reuse can exceed what participants expected.

Data minimization asks what is necessary for the defined purpose. Governance should specify access, audit, and retention. It should specify linkage, permitted reuse, deletion, and response to breaches. Deidentification reduces risk but does not make complex health data harmless. Quality and rights are linked. Data acquired without a durable legal and ethical basis are not a stronger scientific asset merely because they are abundant.

A fit-for-purpose data review#

FDA's Real-World Evidence Framework centers relevance, reliability, and whether study methods can answer the regulatory question. The same logic helps any health-data project.

Map each variable from original event through device, interface, and code. Follow it through warehouse, transformation, and analytic table. Validate critical outcomes and predictors against source records or a reference standard. Measure completeness, timing, duplicates, linkage error, and changes across sites and years.

Assess representation and performance by clinically meaningful groups. Use causal diagrams for etiologic questions and prediction timelines for forecasting. Lock an external or future test set before you start tuning, and decide in advance how you will monitor it and what would make you stop.

When more data truly help#

Additional data help when they add independent outcome events, improve coverage of underrepresented populations, include better reference measurements, capture relevant confounders, extend follow-up, represent new settings, or test temporal transport.

They help less when they duplicate the same patient, repeat the same biased proxy, add post-outcome variables, or expand a population unrelated to intended use. Sometimes the highest-value addition is a small adjudicated validation sample or a qualitative workflow study.

Volume is a resource. Design determines whether it becomes evidence.

References#

1. FDA guidance on EHR and medical claims data
2. FDA Real-World Evidence Framework
3. The Parable of Google Flu
4. Racial bias in a population-health algorithm
5. Racial bias in pulse oximetry measurement
6. Preventing dataset shift from breaking biomarkers