Adopters - Understanding the evidence for a technology before adopting it

When deciding whether to adopt a digital healthcare technology, you need to review the evidence and make sure you understand it.

How to review evidence about a digital technology

Step 1: Checking whether NICE has evaluated the technology

NICE evaluates digital healthcare technologies that address areas of unmet need in health or social care. It reviews the evidence and publishes guidance and advice on these technologies. If NICE has evaluated a technology, this can help you decide whether to adopt it.

NICE guidance

NICE guidance summarises the clinical and cost effectiveness of a technology. It recommends whether the technology should be used in the NHS, and in some cases recommends how it should be used.

You can search NICE guidance and advice to check whether a technology has been evaluated.

Learn more about how NICE evaluates technologies in its health technology evaluations manual.

Early Value Assessment

Early Value Assessment (EVA) produces a faster assessment of a promising technology that is still developing its evidence base but has the potential to meet important needs in health or social care services. EVA recommends whether the technology should be used in the NHS while further evidence is generated.

EVA also outlines the gaps in the evidence base. This is used to develop an evidence generation plan, which NHS adopters can use to guide them in producing real world evidence.

Learn more about EVA and how it is done on NICE’s Early Value Assessment webpage.

Step 2: Reviewing the evidence using NICE’s evidence standards framework

Using NICE’s evidence standards framework (ESF) can help you make informed and consistent decisions about adopting digital healthcare technologies.

Evaluating a digital healthcare technology can be challenging. Using the ESF helps you make decisions about whether to adopt a digital technology. It describes the types and levels of evidence the technology should demonstrate to be adopted by a health or social care service, including standards on security and data governance.

It is particularly important that you review the technology using the ESF if there is no NICE guidance for it. But it is prudent to use the ESF to review the technology even if there is NICE guidance, because:

• digital technologies can change, such as changes to the interface or new features, that might impact the effectiveness as it was originally assessed
• new evidence on clinical and cost effectiveness may have been produced after the guidance was published
• the ESF includes practical guidance for deployment, which will help you during the implementation phase

You can use the ESF to help decide whether to pilot an early version of a technology or include it in an innovation programme.

The ESF is designed for local or regional evaluations of a digital technology. It is intended to stand alongside the regulatory and other standards for digital technologies, not to replace them.

How do I use the ESF?

The ESF has 21 standards for ‘mature’ digital technologies (that is, technologies that have a robust evidence base on safety, clinical and cost effectiveness). These are grouped into 5 areas of the digital technology’s life cycle: design factors, describing value, demonstrating performance, delivering value and deployment considerations.

There are different evidence standards based on the intended use of the technology, and these are grouped into tiers. You need to determine the evidence tier for the technology, then review the appropriate evidence standards.

You can use the ESF to help assess a technology that does not yet have a full evidence base to support its use. There are 16 ESF standards that help you do this.

To start using the ESF, go to NICE’s evidence standards framework for digital health technologies.

Step 3: reviewing the technology against the Digital Technology Assessment Criteria

The NHS Digital Technology Assessment Criteria (DTAC) sets out specific standards for digital healthcare technologies. For a technology to be used in the NHS or social care, it needs to meet the required standards. It is important you understand the assessment criteria when considering whether the digital technology meets these required standards.

The DTAC focuses on 5 core areas:

• clinical safety
• data protection
• technical assurance
• interoperability
• usability and accessibility

The DTAC brings together legal requirements and best practice in these areas. It overlaps with other legal requirements, such as conformity with medical device or data protection regulation.

There is alignment between the DTAC and the ESF, and you will see reference to many of the same standards and regulatory requirements in both. But they differ in focus; the ESF focuses on evidence of clinical and cost effectiveness, while the DTAC focuses on minimum requirements for use in the NHS.

So, it is important you use both the DTAC and the ESF when assessing a technology.

Reviewing the developer’s DTAC responses

The developer should have completed the DTAC by responding to the questions and providing the evidence required. You need to review the developer’s responses and assess whether the digital technology meets the required standards.

It is important that staff reviewing the DTAC have the relevant expertise to assess the digital technology; for example, the clinical safety section should be assessed by a qualified Clinical Safety Officer.

For more information on how developers complete the DTAC, see using the NHS Digital Technology Assessment Criteria.

Step 4: thinking about your public sector equalities duties

When you use a technology in providing care, you need to monitor its effectiveness. This includes its impact on different groups, such as the potential for data-driven technologies to produce biased results that worsen health inequalities. Technologies can also have a positive impact, such as improving experiences for groups who often have a negative experience of health or care services.

Public sector organisations have specific equalities duties that are important to consider when using AI and digital technologies in health and social care.

Before you adopt a technology, you need to think about how using it might impact equalities. Read our guide on the public sector equalities duties to help you think about the monitoring you’d need to put in place if you choose to use an AI or data-driven technology. This can help you make a more informed decision when choosing whether to adopt it.