We've all turned to 'Doctor Google' but how accurate is 'Doctor ChatGPT'?

(Source: PeopleImages/iStock/Getty Images Plus)

Nearly everyone has thought to Google symptoms when attempting to diagnose common medical woes, but what about asking ChatGPT?

Some researchers from Western University set out with the goal of figuring out just how good AI is at diagnosing common illnesses, and whether or not ChatGPT is a reliable resource in health care and medical education.

Schulich School of Medicine and Dentistry assistant professor Dr. Amrit Kirpalani recently published a study which found that ChatGPT was only 49 per cent accurate in diagnosing people.

Although having AI diagnose ailments is definitely not on the table at this time, the study did find that it remains a useful tool for making complex health-care information more digestible for the average person.

“To me, the most relevant finding is that ChatGPT delivered its answers in a very simple and easy-to-understand way,” said Kirpalani. “I think that’s important because you can see the potential for it to be used as a great tool to help people learn and understand medical cases – but it can also be very convincing even when it’s wrong.”

The study provided 150 cases for ChatGPT to diagnose via multiple choice answers, given information like medical history, physical examination results and lab or imaging tests. The paper reported that AI struggled to interpret test results and integrate them into diagnoses, however it was helpful in making medical information more accessible.

At this time, it’s clear that more innovation and advancement is needed before AI can be a useful tool in medicine, however, the study also made it clear that it’s important that people understand the limitations of the new technology.

“AI literacy is important for patients, for providers, for educators and for students because we need to understand how we can use AI responsibly and how it can be applied and leveraged for health care and medical education purposes,” said Kirpalani.

Kirpalani also cautioned that oversight is needed if AI is chosen to be implemented more broadly in medical contexts.