Adopters - Planning for local validation and integrating a digital technology

It is important to integrate and validate digital healthcare technologies before deploying them in a health or care service. Adopters should plan for this during procurement, in liaison with the developer or vendor.

Understanding local validation and integration

Local validation is the activities you do to check a digital healthcare technology will achieve its required performance levels in your health or care service. Digital technologies cannot usually be deployed directly ‘off the shelf’; that is, you need to take your local environment and IT set-up into account. This is a particular consideration for AI and data-driven technology because of dependency on the data it was trained on and how it generalises to your local population. Local validation activities include calibration and longer-term pilot studies. It may take time to fully understand whether the device is performing as it should in your service.

What you need to do before integrating a digital technology

Developers should deliver their healthcare technologies to you with defined performance levels of safety. This is a requirement of the UK Medical Device Regulations 2002. But the technology may not perform perfectly in your specific environment (depending on your infrastructure, culture, work force and local population). Before ‘going live’ (deploying) a technology, you typically need to agree activities with the developer to integrate and validate it.

Local validation activities to make sure performance levels are reached include:

• testing the device using local data sets to make sure processing can occur and within expected performance levels. If possible, assess device performance against specific sub-populations within your local population to check the technology generalises to your needs
• stress testing against your local requirements. This is to make sure the device can function within the expected workflow to meet clinical timelines, and to check for performance issues in the local patient population
• calibrating device parameters to maximise local performance. Note that performance must not drop below safe levels
• doing longitudinal studies or silent-mode testing to explore performance across a wider population sample and understand the impact of drift on performance