Building trust in healthcare, AI and automated decision-making

The AI juggernaut is moving forward at enormous pace. The worlds of finance, insurance, professional services and many, many more are turning to AI and automated decision-making as a way to transform their business processes, to improve efficiency, reduce cost and ultimately make better decisions.   

One area where AI has the potential to achieve some highly beneficial advantages for everyone — especially patients and medical professionals, but also management and operations — is the healthcare sector.

In the UK a recent report by former Labour and Conservative health ministers Lord Darzi and Lord Prior for The Institute for Public Policy Research’s (IPPR) suggested that artificial intelligence (AI) and automation technologies could save the NHS £12.5bn (10% of its annual operational expenses). The report also claims a further £6bn could be saved by automating the repetitive decision making and admin tasks in social care.

ABI Research’s June 2018 report predicts a rapid growth in patient monitoring devices, which include AI for home-based preventative healthcare and predictive analytics, could save hospitals around $52bn by 2021.

Accenture’s Digital Health Technology Vision 2018 report goes so far as to claim that 85% of health executives in the US believe that every human will be directly impacted on a daily basis by an AI-based decision within the next three years.

Showcasing the possibilities

We’re already starting to see a growing number of early examples of AI algorithms and automated decision-making tools being trialled in the healthcare environment, here are some examples;  

On a clinical level Mount Sinai Hospital in New York recently announced a partnership with the AI healthcare start-up RenalytixAI to create an AI tool that identifies patients at the hospital who are at risk of advanced kidney disease. 

Baidu's AI researchers also claim to have built an algorithm that can spot cancerous tumours in breast tissue.

There are numerous news reports being published about Google’s new algorithm. Google claims it is able to quickly sift through thousands of digital documents in patients’ health records to find important information that can help doctors make better predictions about how long patients may stay in a hospital, or the likelihood of when they may die.

And at the aftercare/rehabilitation end of the scale my company has worked with a community healthcare service, Fluid Motion, that delivers group aquatic rehabilitation sessions in local swimming pools for the treatment, management and prevention of musculoskeletal conditions.

We worked closely with Fluid Motion’s qualified physiotherapists and osteopaths to encode their model of patient treatment into our platform, including the suitability of numerous exercises to individual patient symptoms. The resulting AI tool now allows them to create a tailored, illustrated rehabilitation plan for patients as a result of an interaction with a virtual physiotherapist or osteopath.

See also

• Amazon announces its acquisition of online pharmacy PillPack
• Looking past the hype: Getting real-world results from AI in healthcare
• Google looks to utilise AI to further overhaul the GP-patient relationship

Finally, at an administration level, tools like ZocDoc are showing the potential of AI to make it easier for patients and healthcare providers to digitally verify patients’ insurance information. In the US this administrative cost currently makes up about 15 percent of all healthcare expenditures and office managers spend nearly a quarter of their time dealing with insurance.

That’s just the tip of the iceberg.

But there’s a problem

Despite all the positive exuberance for AI in the healthcare sector, an interesting consumer survey by the Royal Society of Arts (RSA) and market research firm YouGov tells a very different story about the public’s general lack of trust in automated decision making. The survey found that in general only 32% of people are aware of automated decision-making systems. More specifically 74% lack any familiarity about the use of these systems to make decisions in healthcare and half (48%) said they were actively opposed to the use of the technology in healthcare.

That’s a lot of fear, scepticism and ‘ignorance’ from arguably the primary stakeholder in any healthcare service.

My interpretation is that much of this opposition stems from science fiction movies and mainstream media scaremongering, but the fact is that many AI and automated decision platforms forgo transparency and are consciously created as black box systems. The techniques used make it very hard to know why a decision or prediction was made.

How can you build public trust and garner support in systems that effectively remain free of legal scrutiny? The truth is that you can’t. While these technologies remain opaque, and operate outside of public view, confusion and distrust will continue.

While the majority of the public take no issue with mainstream consumer AI applications such as Netflix recommendations, auto-tagging on Facebook or suggestions from Siri, it is an entirely different matter when data is used to make healthcare decisions that could potentially be life-altering.

The House of Lords Select Committee on AI has already expressed its view that “…it is unacceptable to deploy any AI system that could have a substantial impact on an individuals’ life, unless it can generate a full and satisfactory explanation for the decisions it will take.”

Trust, safety and compliance in the healthcare AI domain will all rely on technology’s ability to explain the process it used to arrive at decisions. The RSA and YouGov’s research corroborates this from the perspective of the general public whose support for these systems increased by 36% if users could be granted the right to request an explanation of the organisational steps or processes undertaken to reach a decision with an AI system.

We also need to think about the impact of healthcare AI in a much wider context than just the patients alone. Doctors, nurses and NHS leadership also have to trust automated decision making if adoption of the technology is going to happen en-masse. If a consultant is using AI to make a decision on a terminal diagnosis, they need to believe the technology is trustworthy and dependable. If diagnoses are challenged there needs to be an audit trail showing how decisions were reached and the level of certainty the technology had when it recommended the decision.

We need more algorithms for automated decision-making that are modelled on human expertise – rather than black box data. At its heart AI is for collaboration with people. By building human/medical expertise into the development of healthcare AI platforms we can enable these systems to provide clear explanations for the decisions they make. And they can do this in a format that those same human experts understand and can confirm. Human-down AI is critical to the technologies future in healthcare and in building patients’ and healthcare professionals’ trust in automated decision making.

The current disconnect between how automated decision-making works and the public’s understanding of it cannot continue. Transparent, auditable and accountable platforms must be the norm rather than the exception.

If the public at large’s fear and distrust in healthcare AI isn’t properly addressed we risk wasting the enormous potential AI in healthcare has to benefit each and every one of us. We can’t let that happen.