Why Digital Health Tools Survive Routine Care

Key points#

• Start with a defined health or service problem. Technology is one possible intervention, not the objective itself.
• Match evidence requirements to the tool’s function and potential consequence. Usability evidence cannot substitute for effectiveness or safety evidence.
• Design for the whole work system: people, timing, data, handoffs, exceptions, support, and downtime.
• Measure continued value, burden, harm, equity, and abandonment after launch.
• Plan ownership, funding, updates, incident response, and retirement before the pilot begins.

A durable tool has a durable value chain#

A digital health product can be technically available yet functionally absent: staff may bypass it, patients may stop returning, alerts may be dismissed, or a dashboard may stay open without changing a decision. Counting installations or logins misses this distinction.

Durability is a chain of conditions. A meaningful need has to be matched to an appropriate digital function. The function must have evidence for its intended claim. It must fit the people and workflow that deliver it. Data and infrastructure must remain dependable. The organization must be able to govern, support, update, and pay for it. Finally, measurement must show that value persists without unacceptable burden or inequity.

Break any important link and continued use becomes fragile. The WHO guideline on digital interventions makes a related point: digital tools do not replace the foundations of a functioning health system. They operate within those foundations and inherit their limits.

Test the problem before the product#

Begin with a problem statement that does not name a technology. Specify who encounters the problem, when it occurs, what the current process is, and which outcome needs to improve. “Clinicians need an app” is a proposed solution. “Medication reconciliation errors occur during transitions because two lists are not compared before discharge” is a problem that can be investigated.

Map the current process, including workarounds. Some apparent information gaps are actually staffing, authority, scheduling, or communication problems. Digitizing a poorly defined process can make its defects faster and harder to see.

Then state the proposed mechanism. A reminder may work by making a due task visible. A risk estimate may work by prioritizing review. Remote monitoring may work by detecting change between visits. Each mechanism requires different data, evidence, timing, and response capacity. A signal without someone able to act on it is not a complete intervention.

Match evidence to the claim#

Evidence should become more demanding as a tool moves from organizing information to influencing consequential decisions, and NICE’s Evidence Standards Framework classifies digital health technologies by function and links those functions to appropriate evidence expectations. The principle is more important than any single jurisdictional category: evaluate the claim being made, not the sophistication of the code.

Several layers may be needed:

• Technical performance: Does the system process inputs and generate outputs as specified?
• Usability: Can intended users complete the task accurately, including under time pressure and with accessibility needs?
• Analytical or model performance: Does an algorithm perform on representative, independent data using suitable metrics?
• Clinical or service effectiveness: Does using the tool change a meaningful process or outcome compared with a relevant alternative?
• Economic value: Are benefits proportionate to acquisition, integration, training, support, and opportunity costs?
• Distribution of effects: Do performance, access, burden, and outcomes differ across relevant groups or sites?

A short pilot can establish feasibility. It rarely establishes long-term effectiveness, safety, or affordability. Early adopters may receive intensive support that will not exist after scale-up. Outcome evaluation should preserve that distinction.

Fit the moment of work#

AHRQ describes effective clinical decision support as the right information for the right people, in the right format, through the right channel, at the right point in workflow; the phrase is useful because it directs attention away from the model alone.

Observe the intended users doing the actual task, and identify what information is available at that moment, what decisions have already been made, what other screens are open, and what happens after the output appears. Include administrative staff, patients, caregivers, interpreters, and technical teams when they carry part of the process.

Count more than clicks. A new screen may require authentication, duplicate data entry, reconciliation of conflicting values, documentation, and follow-up. Five seconds of computation can generate several minutes of human work. Conversely, a well-placed default can reduce steps while preserving an obvious way to reconsider the choice.

Exceptions reveal design quality. Define what happens when data are missing, the patient falls outside intended use, or the user disagrees. Define what happens when the network fails or an alert is not acknowledged. Safe fallback should be deliberate, visible, and tested.

Build the operating system around the software#

The NASSS framework examines the condition, technology, and value proposition. It examines adopters, organization, wider system, and adaptation over time. Its central lesson is that sustainability is a sociotechnical property. Complexity can accumulate across domains even when each technical component works.

Before you launch, assign named ownership for:

• clinical content and evidence updates;
• data definitions, interfaces, and quality checks;
• privacy, security, consent, and records management;
• accessibility and language access;
• training, support, and incident triage;
• algorithm or rule performance;
• contracts, licensing, and recurring cost;
• version approval, communication, rollback, and retirement.

An interface change in a source system can alter a field without crashing the tool. A guideline update can make an encoded recommendation obsolete. A vendor update can change behavior at a time chosen outside the clinical organization. These are operational risks, not rare edge cases.

Interoperability also involves meaning. Two systems may exchange a value while using different units, reference intervals, timestamps, or definitions. Validation must cover the local data path from source to display, not only successful message transmission.

Measure continued value, not launch activity#

The WHO monitoring guide separates implementation fidelity from impact. Both matter. If an outcome does not improve, the question you face is whether the underlying idea failed or the intervention never reached intended users in its planned form.

A practical measurement set can include:

• reach among eligible users and patients;
• completion of the intended action, not just opening the tool;
• time, clicks, interruptions, and support requests;
• overrides, ignored outputs, and documented reasons;
• data completeness and system availability;
• process and outcome measures linked to the claim;
• adverse events, near misses, complaints, and privacy incidents;
• performance and access across relevant groups and sites;
• cost of ownership, including human maintenance;
• continued use and reasons for nonuse or abandonment.

Choose a comparator and a baseline. A rising use count after mandatory rollout says little about whether the prior process improved. Balance measures are also necessary: a reminder may increase completion while adding low-value testing or staff burden elsewhere.

Qualitative evidence explains numbers. Interviews, observation, and support logs can show why people override an output. They can show which work has shifted to another team, or where a technically correct recommendation collides with local constraints.

Adapt without losing control#

Sustainable tools change because evidence, populations, workflows, and technology change. Uncontrolled change, however, can invalidate earlier evidence. Keep a version history that connects each modification to its rationale, risk assessment, and testing. The history also connects approval, release date, and monitoring plan.

Distinguish content updates from interface changes, data-pipeline changes, and model changes. Each has a different failure pattern. Test in conditions that represent intended use, notify affected users, preserve rollback where feasible, and reassess the original claim when a modification is substantial.

Retirement is part of stewardship. Define triggers such as loss of evidence support, unacceptable burden, or unresolved performance decline. Other triggers are unavailable maintenance, replacement by a safer process, or persistent inequity. A tool should not continue merely because removing it lacks an owner.

The sustainability question#

Ask yourself one concrete question twelve months after launch: can you show that the tool still reaches the intended people, behaves as expected, improves the defined problem, and remains supportable? A defensible answer needs operational records and outcome evidence, not enthusiasm from the original rollout.

Articles on model drift and monitoring and appraising clinical prediction models examine two parts of that continuing evidence obligation in greater depth.