Can AI Become Our Trusted Clinical Co-pilot?
By John Lorinc
Artificial intelligence promises to reshape health care as we know it, saving precious time — and lives.
Medical researchers and clinicians are at the forefront of one of the most dynamic — and sometimes contentious — fields in technology. But as we harness the power of AI in everything from diagnostics and imaging to chatbots for patient interactions, what’s ahead for this bold new partnership between humans and machines?
In 2020, a Unity Health Toronto team launched an AI-driven software platform designed to be the digital eyes and ears of the physicians and nurses in two busy hospitals. CHARTWatch tracked about 150 data points, including vital signs, test results and demographics flowing into the electronic health records of patients admitted from the ER to a general internal medicine ward. But the algorithm embedded in CHARTWatch not only watched that data, it continually analyzed it, looking for patterns that indicated an escalating emergency situation and, when necessary, alerted medical staff.
Earlier this fall, CMAJ published the results of a study that compared the results of CHARTWatch’s first two years on call with the three previous years. The findings confirm the efficacy of a technology specifically intended to cut the risk of in-patient deaths. The study found that when CHARTWatch was functioning, there were “significantly lower” non-palliative deaths and a 26 per cent reduction in unexpected mortality.
Muhammad Mamdani, the study’s senior author and the Unity Heath’s vice-president of data science and advanced analytics, says CHARTWatch seemed particularly well suited to spotting previously undetected infections, such as sepsis that requires antibiotics, and worsening cardiac conditions.
One of the key findings, he explains, is that the algorithm inside the platform tends to be more accurate than physicians at capturing the untimely deaths of very sick patients admitted to hospital wards.
“What we’ve learned is that clinicians are typically not very good at prognosis,” says Mamdani, who is also the director of the Temerty Centre for Artificial Intelligence Research and Education in Medicine (T-CAIREM) at the Temerty Faculty of Medicine.
“We collected over 3,000 clinician predictions and were able to show that when a physician says a patient’s going to die or go to the ICU, they’re right less than a third of the time. That told us that the AI algorithm doesn’t have to be perfect, but it has to beat a very low bar.”
The Promises, the Risks
The envisioned benefits run the gamut, from improved clinical assessments of neurological pain to effectively managing day-to-day tasks, such as documenting patient encounters with primary care doctors who use AI scribe technology for note-taking.
We constantly have to educate our clinicians, new residents, new nursing staff and new physicians — this is a collaboration between AI and humans
Yet, as with all things AI, there are red flags including the known racial biases of machine learning, mistakes in the vast data sets used to train the algorithms and nonsensical outputs, which the AI industry calls “hallucinations.” The technology also poses important questions about safety, efficacy and the regulation of AI-driven software systems embedded in medical devices.
Human factors engineer Enid Montague, an associate professor in the University of Toronto’s Department of Mechanical and Industrial Engineering, says that the goal should be to find a balance between these technologies and the people they serve, including both clinicians and patients, and particularly those from marginalized groups in society.
“There is an excitement to say, ‘Let’s just automate everything in health care,’” Montague says. “But I’m really interested in patients being able to use … generative AI tools to help them manage the work that they have to do themselves in decision-making.”
Expanding Possibilities for Patient Care
One of the most promising applications of AI technology in health care involves imaging and using the software to support both diagnostics and patient interaction. Mojgan Hodaie (MSc’94, PGME’04), the surgical co-director of the Joey and Toby Tanenbaum Gamma Knife Radiosurgery Centre at University Health Network’s Toronto Western Hospital, has developed an AI application to support her neurosurgery practice.
Hodaie, who is also a surgery professor at Temerty Medicine, was familiar with early imaging-related AI applications, to interpret X-ray results. But she saw an opportunity to apply AI to the diagnosis of trigeminal neuralgia, a condition marked by intense attacks of facial pain that typically affects older people. It results from a blood vessel compressing a nerve in the brain stem.
She knew that MRI didn’t always show nerve compression in patients presenting with symptoms. “We wanted to know: What else can we look at in the brains of these patients to give us a better understanding of this pain condition?”
Her team built an algorithm trained on thousands of brain scans conducted at Toronto Western and even larger sets of publicly available brain MRIs. The team sought to identify a specific “pain signature” — small structural changes in the brain, such as an abnormal thickness in specific grey matter areas, that occurred in patients with known trigeminal neuralgia. The algorithm scanned the MRIs for a proxy, seeking the specific neurological fingerprints the condition leaves, to provide additional certainty when diagnosing patients whose imaging does not show evidence of the tell-tale nerve compression.
Hodaie’s team hasn’t rolled out this application in a clinical setting; however, with the algorithm, she will be able to tell her patients with far more precision the likelihood of a surgical intervention being successful. She also envisions applications for other elusive pain conditions in which a specific physical symptom, such as a tumour or broken bone, isn’t discernable using conventional imaging.
“I fundamentally think, in 2024, we really need to do better for our patients,” Hodaie says. “I should have a better opportunity of telling them how likely it is that I will be able to help them, and what their experience in care will involve.”
Not all health care related AI applications drill as deeply into medical processes. Mamdani points to examples such as AI-supported scheduling platforms that allow ER administrators to ensure that nurses, physicians and other health care staff are deployed according to hospital policies; for example, to cover routine rotations between units or to pair junior and senior staff.
“It typically takes a senior nurse about 90 minutes a day to assign nurses to different zones in the emergency department. A clerk on our medical staff spends about two hours a day on this task,” Mamdani says. But this human approach, he notes, violates HR policies 20 per cent of the time.
Conversely, he says, the AI scheduler “looks at the nurses’ history, it looks at the rules and then assigns the nurses.” The time spent on this task plummeted to a matter of minutes, with far fewer violations of hospital HR policies.
Evolving Governance
Jay Shaw, an assistant professor of physical therapy at Temerty Medicine and the research director of AI, Ethics & Health at U of T’s Joint Centre for Bioethics, points out that the governance of such technologies is crucial, and an evolving domain.
Tech firms develop health care related AI applications that may or may not be subject to approval by Health Canada or the United States Food and Drug Administration. Others, such as CHARTWatch and Hodaie’s MRI-based algorithm, emerge from academic research labs that are subject to university research ethics protocols. Shaw notes that most of the health care related AI-based applications he has seen are not classified as medical devices and so are exempt from the kind of regulatory scrutiny that applies to manufacturers of heart valves or blood thinning drugs.
Shaw also points out that experience from non-health care AI technologies shows how applications have been developed without input from the individuals who will be affected by their use. Community engagement and a rigorous approach to addressing ethical issues will make these technologies better and more effective, he says, pushing back against the notion that ethics is an impediment.
If you don’t get the ethics right,
it’s going to come back to bite you
A multidisciplinary approach is similarly crucial. Shaw cites a project, funded by the Canadian Institute for Advanced Research, to develop AI tools for diabetes prevention. “That project is about a very specific, narrowly defined algorithm that predicts the onset and complications related to type 2 diabetes,” Shaw says, adding that the team includes machine learning experts, epidemiologists, ethicists as well as members of the community. “It’s important to talk to people who are affected by this.”
Transparency and Safety
These concerns will grow more important as generative AI technology becomes ever-more powerful and the potential applications multiply. The most recent versions of large language models (LLM) like ChatGPT, for example, score well on medical school exams and can provide detailed answers to patient prompts. And some researchers are looking into the possibility of allowing patients to interact with specialized LLMs as a first interaction with care.
As University of California San Diego virologist Davey Smith told JAMA, “You could imagine that people who don’t have access to primary care providers all the time, or highly specialized physicians such as myself, might be able to still access information in appropriate ways to receive some health care and health care guidance.”
But in a recent issue of Nature, a U of T team led by Bo Wang (MSc’12), chief AI scientist with UHN and an assistant professor with Temerty Medicine’s Department of Laboratory Medicine and Pathobiology, argues that the researchers and clinicians building these emerging applications should use open source LLMs. Using non-proprietary algorithms, for example, those developed by tech giants such as OpenAI, would ensure transparency and safety.
“In the rush to deploy off-the-shelf proprietary LLMs,” the researchers caution, “health care institutions and other organizations risk ceding the control of medicine to opaque corporate interests. Medical care could rapidly become dependent on LLMs that are difficult to evaluate, and that can be modified or even taken offline without notice should the service be deemed no longer profitable — all of which could undermine the care, privacy and safety of patients.”
A Powerful Partnership
Geoffrey Hinton, 2024 Nobel Laureate in physics lauded as the “Godfather of AI,” has sounded the alarm on the perils of the technology outsmarting its human creators. But the professor emeritus in U of T’s Department of Computer Science also extols its promise, specifically for health care applications.
“AI can make a tremendous difference” in diagnosing difficult cases, Hinton told journalists at a U of T news conference in honour of the Nobel announcement, citing data about the improvement in accuracy when humans and machines work together.
From Mamdani’s vantage point at the intersection of AI-driven research and clinical care, he is keenly aware of the risks, such as physicians or nurses becoming overly reliant on systems that generate diagnoses or catch tell-tale indicators that a patient is in decline.
He refers back to a lesson from the deployment of CHARTWatch. “We constantly have to educate our clinicians, our new residents, new nursing staff and new physicians that this is a collaboration between AI and humans,” he says. “It’s not one or the other — it’s the two working together.” •