Adopters - Piloting digital technologies in a health or care service

Before deciding whether to adopt a digital healthcare technology, you may need to pilot it in your service.

Understanding local piloting

Before adopting a digital healthcare technology, you should have evidence showing its clinical and cost effectiveness in a setting like yours. If evidence relevant to your context is limited, the developer should have plans to generate it. Testing the technology in your service (local piloting) may be part of this plan.

You might do a pilot study for other reasons, including to:

• learn about potential risks or technical and implementation issues associated with the technology in a controlled way before ‘going live’ (deploying) it
• understand how the technology affects current care or operational workflows, and whether this creates risks or issues that need to be managed
• learn how to use the technology most effectively and adjust processes to get the best value from it
• plan processes for monitoring the technology after it is deployed

Piloting gives stakeholders confidence that the technology creates benefits in your setting, and that any decision to adopt it is well-evidenced.

Planning a pilot study

Your pilot study should have:

• a defined scope
• a data-collection plan
• clear criteria for success or failure
• a defined time period (start and finish dates)
• defined roles and responsibilities for you and the developer so that both your interests are met appropriately

The pilot study should generate results that help you decide whether to continue using the technology. So, your data-collection plan should define what and how much data to collect and how it will be analysed. This will allow you to assess the benefits of the technology. Types of data you might collect include:

• healthcare professionals’ experiences of using the technology
• experiences of patients or service users
• clinical or care outcomes
• cost effectiveness
• efficiency gains or losses
• operational or financial impacts on your service

Resources to help you design a pilot study

Resources that may help you design a pilot study include:

• NICE’s real-world evidence framework. This provides general guidance on planning, doing and reporting real-world evidence studies. This will help because you’ll be doing the pilot study in a real-world setting, using data collected routinely in your service
• The DECIDE-AI reporting guidelines, which aim to improve reporting of studies
• NICE’s evidence standards framework for digital health technologies. This provides standards on evidence requirements for different types of digital healthcare technologies (which it refers to as DHTs). See the table below for evidence standards that are particularly relevant for thinking through different stages of pilot development

Relevant NICE evidence standards for developing a pilot

Planning a pilot study

Make sure the value proposition for how the technology will fit into local clinical pathways and its expected benefits are clear:

• Standard 10: describe the intended purpose and target population
• Standard 11: describe the current pathway or system process
• Standard 12: describe the proposed pathway or system process using the DHT
• Standard 13: describe the expected health, cost and resource impacts compared with current care or system processes

Planning for deployment

Make sure the technology deployment plan for the pilot provides relevant information about deployment requirements:

• Standard 19: ensure transparency about requirements for deployment
• Standard 20: describe strategies for communication, consent and training processes to allow the DHT to be understood by end users

Clinical or care outcomes

Identify the data collection requirements to demonstrate the relevant outcomes:

• Standard 14: provide evidence of the DHT's effectiveness to support its claimed benefits
• Standard 15: show real-world evidence that the claimed benefits can be realised in practice

Experiences of patients or service users

Identify the data collection requirements to demonstrate the relevant outcomes:

• Standard 14: provide evidence of the DHT's effectiveness to support its claimed benefits
• Standard 15: show real-world evidence that the claimed benefits can be realised in practice

Experiences of health care professionals

Identify the data collection requirements to demonstrate the relevant outcomes:

• Standard 14: provide evidence of the DHT's effectiveness to support its claimed benefits
• Standard 15: show real-world evidence that the claimed benefits can be realised in practice

Efficiency gains or losses and their operational or financial impact on your service

Identify the data collection requirements to demonstrate the value of the technology:

• Standard 17: provide a budget impact analysis
• Standard 18: for DHTs with higher financial risk, provide a cost-effectiveness analysis