Could ChatGPT soon be used as a diagnostic tool?

(Source: Dilok Klaisataporn/iStock/Getty Images Plus)

A research team in London is looking into whether or not ChatGPT can be used as a diagnostic tool for medical learners and clinicians.

Dr. Amrit Kirpalani, a paediatric nephrologist at Children’s Hospital at London Health Sciences Centre, and his team of Schulich Medicine students, wanted to explore how ChatGPT could help learners and professionals.

To do so, they inputted 150 medical case challenges into ChatGPT, and the answers were analyzed based on accuracy, relevance and cognitive load.  

According to Kirpalani, the results concluded that ChatGPT 3.5 accurately diagnosed 49 per cent of the 150 case challenges, but that the overall reliability was limited.

The team found that ChatGPT struggled with interpreting numerical values, neglected data points key to diagnoses and often overlooked important details which led to inaccurate assumptions.

However, the team was interested in ChatGPT’s ability to take complex cases and explain answers in a way that is understandable to those who may not possess complex medical knowledge.

“The cognitive load was relatively low, meaning that it was easy to understand,” said Branavan Nagarajan, a third-year Schulich Medicine medical student involved in the project. “This means ChatGPT can be a powerful tool for the learning process, helping both pre- and early-year medical students understand complex concepts because they are presented in a more manageable format. But when you combine this with the issues of accuracy, it highlights the importance of having someone with a higher level of medical knowledge to validate the outputs.”

Going forward, the research team believes there are exciting opportunities in terms of next steps.

“We are curious to analyze the performance of future iterations of ChatGPT, to see how much better it does in terms of relevancy, accuracy and cognitive load,” said Nagarajan. “There could be the opportunity to expand the parameters the program receives, allowing us to incorporate more numerical data, images and even clinical content such as patient history. We would be curious to see how this could impact the program’s diagnostic accuracy.”

Kirpalani said he would like to explore the potential of incorporating ChatGPT and AI literacy into formal medical training, adding, “Our results illustrate the importance of understanding how to use these AI tools properly to avoid receiving or spreading misinformation. A tool is only powerful if we know how to use it accurately.”