The Difference Between Validation and Marketing

“Clinically validated” can mean a rigorous prospective study of a locked product in the intended workflow. It can also mean that a company compared an output with something once, and the phrase is worth nothing to you until you can see the claim, the product version, the population, the reference, the endpoint, and the study design.

Validation is a scientific and engineering activity. It asks whether a specified system meets specified requirements for a specified use. Marketing is communication intended to shape attention or choice. Good marketing can summarize validation accurately. Poor marketing can stretch a narrow result into a broad promise without making a statement that looks obviously false in isolation.

Your job as a reader is to rebuild the bridge from the study to the claim, one plank at a time; every widened population, every stronger verb, every different endpoint, and every changed product version adds an inference that somebody has to support.

Start with a claim that could fail#

“Our AI improves health care” is too broad to validate. “In adults referred for diabetic retinopathy screening at primary-care clinics, version 3.2 identifies more-than-mild disease from gradable fundus images at the prespecified sensitivity and specificity thresholds compared with a masked reference panel” is testable.

A useful claim identifies:

1. Product and version.
2. Intended user and setting.
3. Target population and important exclusions.
4. Input and output.
5. Comparator or reference standard.
6. Endpoint and threshold.
7. Time horizon where relevant.
8. Required level of performance.

Validation then tests that claim with a protocol and acceptance criteria chosen before seeing the final results; a study can be informative without passing every criterion, but a failed criterion should not disappear from the public story. Marketing often begins with the phrase the audience will remember and places conditions elsewhere. Evidence appraisal begins with the conditions.

Verification and validation are related but different#

Verification asks whether the system was built according to its design requirements. Does the software calculate correctly? Do interfaces pass inputs without corruption? Does the sensor meet its analytical specification? Do risk controls operate as designed?

Validation asks whether the finished system meets user needs and intended use in the relevant context, because a perfectly coded algorithm can be clinically unhelpful because the target is wrong, the population differs, the output arrives too late, or users misunderstand it.

For software, unit tests and code review support verification. They are not clinical validation. For a laboratory assay, precision and linearity are analytical evidence. They do not by themselves establish that using the result improves a decision. Marketing can blur these layers by calling any technical test “clinical validation.” Ask which layer was actually examined.

Internal validation estimates optimism#

A prediction model usually performs better on data used to develop it than on new data, and internal validation estimates and adjusts for this optimism through methods such as bootstrapping, cross-validation, or a well-designed resampling process.

A random train-test split is not automatically strong internal validation. A small dataset loses development information when split. Records from the same patient, site, device, or time series can leak across partitions. Repeated tuning against the test set turns it into development data.

Internal validation is necessary during development, but it cannot establish that the model works at another institution or a future time, because the data share so many collection, labeling, prevalence, and workflow characteristics. So when a model is described as “validated on 100,000 records,” find out whether those records are a held-out slice of the same source. Size does not repair dependence or leakage.

External validation tests a new context#

External validation evaluates the fixed model on data separated from development in a meaningful way. Separation can be geographic, temporal, institutional, technical, or population-based.

The model should be locked before evaluation. The study reports discrimination, calibration, and threshold-specific performance. It reports missing inputs, invalid outputs, and uncertainty. Performance should be assessed for clinically relevant subgroups without pretending small samples give precise estimates.

One external validation is still one setting. A tool can transport across two academic centers and fail in community practice. That happens because prevalence, documentation, or equipment differ. It also happens when staffing or referral patterns differ.

If the developers recalibrate or retrain the model using the new site, the exercise becomes model updating. The updated version needs its own separated evaluation. The TRIPOD+AI statement provides reporting guidance for prediction model development and validation regardless of whether regression or machine-learning methods are used.

Diagnostic accuracy requires a trustworthy reference#

A diagnostic study compares an index test with a reference standard for a defined target condition. If the reference is imperfect, biased, or influenced by the index result, apparent accuracy can mislead.

You need to know who established the reference, whether they were masked, and how disagreements were resolved. You need to know how indeterminate results were handled and whether every participant received the same reference. Differential verification can bias estimates when positive and negative index results receive different confirmation.

Case-control designs that compare obvious disease with healthy controls can exaggerate performance because real clinical cases include mild, early, atypical, and competing conditions. The intended-use population should determine recruitment. The 2025 STARD-AI guideline makes AI-specific reporting expectations visible. Reporting compliance is not proof of quality, but missing details become harder to hide.

Analytical validity, clinical validity, and utility form a ladder#

Analytical validity asks whether a test accurately and reliably measures the analyte, variant, signal, or feature. Precision, bias, and range belong here. So do interference, stability, and reproducibility.

Clinical validity asks how well the result identifies or predicts a clinical state or outcome in the intended population. Sensitivity, specificity, calibration, risk ratios, and predictive values may be relevant.

Clinical utility asks whether using the result improves decisions or outcomes enough to justify harms and burden. A genomics score can predict disease without changing what clinicians should do. An imaging algorithm can find more lesions while increasing unnecessary procedures. You cannot skip a rung: utility usually requires analytical and clinical validity, but strong validity does not deliver utility on its own. Marketing commonly climbs from “correlated with” to “detects,” then from “detects” to “prevents.” Each verb names a different rung.

Prospective evaluation tests real inputs and timing#

Retrospective data are efficient and valuable. They often omit workflow failures that occur when a system runs live: missing inputs, latency, and user overrides. Others are connectivity, changed behavior, and feedback loops.

A prospective silent study runs the tool on incoming data without showing outputs to clinicians. It can measure local performance, uptime, and alert volume without changing care. It cannot show how people respond.

An early live evaluation shows outputs to intended users and measures usability, workflow, human factors, and safety. The DECIDE-AI guideline addresses reporting of this stage.

An impact study compares care with and without the tool, ideally using randomization when feasible. Outcomes may include appropriate treatment, time to action, and complications. They may include patient experience, workload, and cost. The relevant unit might be patient, clinician, clinic, or hospital. If a page says “prospectively validated,” find out which of those designs it means. Prospective data collection alone does not prove comparative benefit.

Performance metrics answer different questions#

Accuracy is the proportion of all classifications that are correct. It can look excellent when the target is rare and a model predicts almost everyone negative.

Sensitivity and specificity describe performance among reference-positive and reference-negative cases. Positive predictive value describes how many positive calls are correct and changes with prevalence. Negative predictive value has the corresponding dependence.

Area under a receiver-operating curve summarizes ranking across thresholds. It does not specify calibration, workload, or clinical benefit at the chosen threshold. Calibration asks whether predicted probabilities match observed event frequencies.

Decision-curve or net-benefit analysis connects true and false positives to a threshold, but it still depends on appropriate consequences and assumptions. Time-to-event models need assessment across time and competing risks. A marketing claim can select whichever metric looks largest. Validation predefines the metric that matches the intended decision and reports the rest needed for interpretation.

Denominators and units can manufacture impressions#

“95% accurate” could be per image, per lesion, per patient, per visit, or per alert, and a model can have few false marks per image and many false marks per examination because each examination contains many images.

Excluding ungradable inputs raises apparent accuracy if failure to produce a result is common in practice. Missing participants after enrollment can change the population. A per-protocol analysis may describe ideal use while hiding failures in deployment.

Absolute counts are what clear this up, so ask for them: true positives, false positives, and true negatives. Ask for false negatives, invalid outputs, and missing outcomes, each per meaningful unit. Relative improvement also needs an absolute baseline. A 50% reduction from two events to one per 10,000 differs from a reduction from 20 to 10 per 100.

The product version is part of the evidence#

Software can change faster than a paper is published. New training data, architecture, or threshold can alter performance. So can preprocessing, user interface, sensor, or label.

A validation belongs to the tested version and configuration, so a company should map each public claim to a version history and explain whether an update was covered by the evaluation, bridged analytically, or newly validated.

For regulated AI-enabled devices, an authorized predetermined change control plan can describe certain planned modifications and validation methods. It does not make every future change prevalidated. Model drift can occur without a code change because patient mix, equipment, clinical practice, and data pipelines change. FDA research on postmarket monitoring focuses on detecting input and output performance changes across use.

Human-system performance is not model performance#

A decision-support system works through a person and organization. Users can ignore correct outputs, follow incorrect outputs, misunderstand uncertainty, or change their information search after seeing a recommendation.

Automation bias can increase acceptance of wrong suggestions. Alert fatigue can reduce response to useful ones. A tool can move work from physicians to nurses without changing total burden, or improve speed while fragmenting communication.

The FDA, Health Canada, and MHRA transparency principles emphasize intended users and context. They emphasize workflow, limitations, and the human-AI team. A claim that an algorithm “matches specialists” in a standalone test does not establish that nonspecialists using it match specialists, or that specialists assisted by it improve patient outcomes.

Subgroup performance needs both purpose and power#

Models can perform differently by age, sex, and skin tone. They can perform differently by disability, language, and ancestry. They can perform differently by disease severity, site, and device. Aggregate performance can hide clinically important failure.

Subgroups should be chosen based on intended use and plausible mechanisms. Sample size and event count determine uncertainty. A point estimate from 12 cases cannot support a confident equity claim.

Equal sensitivity does not guarantee equal benefit if prevalence, access to confirmation, or treatment differs. Calibration can vary even when discrimination is similar. Marketing images can imply universality while the study population was narrow. Validation tables should show who was included, who was missing, and where uncertainty is large.

Regulatory status does not replace claim analysis#

FDA registration, establishment listing, or a product listing is not marketing authorization. A 510(k) is cleared, a De Novo request is granted, and a PMA is approved. Some software functions are not devices, and some fall within enforcement policies.

Even an authorized device has a specific indication, intended user, setting, and label. An advertisement can exceed the reviewed claim by implying autonomous use, broader populations, superior outcomes, or prevention.

Peer review asks whether a manuscript meets a journal's standards. It is not product authorization or replication. A preprint is public but not peer-reviewed, and a conference abstract often contains limited methods. Awards, patents, and investor funding sit further away still: they show novelty or commercial interest, not clinical performance.

Testimonials and “real-world” stories#

A testimonial can be genuine and unrepresentative. Outcomes may reflect selection, placebo response, simultaneous care, natural history, or regression to the mean. A dramatic case cannot estimate a rate.

“Real-world evidence” describes data or evidence arising from routine settings under defined methods. It does not mean any customer story, dashboard, or uncontrolled before-after comparison.

Before-after data are vulnerable to secular trends, changing eligibility, learning, and concurrent interventions. A comparison group and prespecified analysis can improve inference.

The FTC Health Products Compliance Guidance says marketers should consider the express and implied messages consumers take and ensure that support is scientifically sound, adequate in the surrounding evidence, and relevant to the specific product and claim.

Fine print cannot always cure the headline#

An advertisement communicates through headline, images, and layout. It communicates through testimonial, audio, and omission. “May support” in small text may not correct a prominent promise that a product prevents disease.

Disclosures should be clear and close to the claim. A limitation buried behind a link may not change the net impression. Stating “results may vary” does not substitute for describing the study population and outcome.

Claims of equivalence need an appropriate margin and design. “Noninferior” is a formal statistical conclusion tied to a prespecified margin, not a synonym for similar-looking percentages. Marketing can legitimately simplify. It cannot change the answer to what was tested.

A claim-to-evidence audit#

For any health or AI claim, build a short table with these columns:

• Exact public wording.
• Implied action or outcome.
• Product version and label.
• Supporting study and design.
• Population, setting, and comparator.
• Endpoint, threshold, and absolute counts.
• Important exclusions and limitations.
• Whether the evidence is internal, external, prospective, or an impact evaluation.
• Whether current monitoring still supports the claim.

A gap in one row is not always deception. It is a question you should get answered before you rely on the thing. The audit is especially useful when one study supports several web pages, sales decks, and press statements. Claim drift can occur even when the underlying study stays the same.

Warning signs of validation theater#

Look closely when a company provides a large participant count but no data provenance, says “peer reviewed” without a citation, reports only AUROC, or calls a random data split external validation.

Other warning signs include changing the threshold after seeing results, excluding invalid outputs from the denominator, reporting a best subgroup as the whole population, comparing with no meaningful baseline, and using the same data for tuning and final testing.

“FDA registered,” “patented,” “used by leading institutions,” and “built by doctors” do not answer performance. “No adverse events reported” is uninterpretable without active monitoring, denominator, and follow-up. The strongest signal is not polished confidence. It is enough detail for another qualified team to test the claim.

What credible communication looks like#

Credible product communication names the intended use, version, and study design. It names population, setting, and comparator. It names the main endpoint, absolute results, and uncertainty. It names limitations and regulatory status.

It distinguishes research use from clinical use, development from external validation, association from benefit, and technical failure from a negative result, and it states who should not use the product and what happens when the output is invalid.

Updates correct prior claims when evidence changes; monitoring criteria and known failure modes are public enough for users to manage risk, and the sales training uses the same boundaries as the labeling and the scientific materials. None of that costs a company its value proposition. It is how the evidence gets to carry the message instead of the adjectives.

The evidence-matched conclusion#

Validation is not a badge. It is a bounded result from a defined evaluation. Marketing is not inherently opposed to science, but it creates pressure to remove the boundaries that make the result true.

The solution is traceability. Every claim should map to the product version, intended use, study, population, endpoint, threshold, and current monitoring. Each stronger verb should require stronger evidence.

When the bridge is visible, you can decide for yourself whether the product was merely tested, predicts accurately, works in practice, or improves outcomes; those are four different achievements, and honest marketing keeps them different.

References#

• Collins GS, et al. TRIPOD+AI statement. BMJ. 2024.
• Sounderajah V, et al. STARD-AI reporting guideline. Nature Medicine. 2025.
• Vasey B, et al. DECIDE-AI early clinical evaluation guideline. Nature Medicine. 2022.
• FDA, Health Canada, and MHRA. Transparency for machine-learning-enabled medical devices.
• International Medical Device Regulators Forum. Good Machine Learning Practice guiding principles. 2025.
• Federal Trade Commission. Health Products Compliance Guidance. 2022.