ChatGPT Accuracy and Health Information Risks

By Nithya Somarapu on April 4, 2024

ChatGPT accuracy study highlights challenges in AI reliability for health-related information.

According to a groundbreaking study on ChatGPT accuracy, the AI model gets less trustworthy the more evidence is provided when asked about a health-related topic; its accuracy of response can drop as low as 28%.

The research was recently presented at the Empirical Methods in Natural Language Processing (EMNLP) conference. The results have been published in the Proceedings of the Empirical Methods in Natural Language Processing Conference, 2023.

Large language models (LLMs) such as ChatGPT are becoming more and more popular, and this could be dangerous for the increasing number of individuals who rely on online resources for critical health information.

Researchers from The University of Queensland (UQ) and CSIRO, Australia’s national science agency, investigated a fictitious situation in which a layperson (a non-professional health consumer) uses ChatGPT to inquire about the efficacy of “X” treatment for ailment “Y.”

From “Can zinc help treat the common cold?” to “Will drinking vinegar dissolve a stuck fish bone?” were among the 100 questions posed.

The response from ChatGPT was contrasted with the accepted response, or “ground truth,” which is based on current medical understanding.

Dr. Bevan Koopman, an associate professor at UQ and Principal Research Scientist at CSIRO, said that despite the extensively reported risks associated with searching for health information online, people still do so, and are increasingly doing so through ChatGPT and other similar platforms.

“The widespread popularity of using LLMs online for answers on people’s health is why we need continued research to inform the public about risks and to help them optimize the accuracy of their answers,” Dr. Koopman said. “While LLMs have the potential to greatly improve the way people access information, we need more research to understand where they are effective and where they are not.”

Two question formats were examined in the study. The first was only a query. The second question was slanted in favor of or against the evidence.

With an accuracy rate of 80% in this scenario, the results showed that ChatGPT accuracy was fairly good at providing precise responses in a question-only format.

Nevertheless, accuracy dropped to 63% when the language model was presented with an evidence-biased cue. The accuracy dropped to 28% once more when an “unsure” response was permitted. This result defies the widely held notion that providing evidence when prompting enhances accuracy.

“We’re not sure why this happens. But given this occurs whether the evidence given is correct or not, perhaps the evidence adds too much noise, thus lowering accuracy,” Dr. Koopman said.

Since its November 30, 2022 introduction, ChatGPT accuracy has grown to rank among the most popular large language models (LLMs). Artificial intelligence in the form of LLMs can recognize, translate, summarize, forecast, and create text.

Major search engines are increasingly merging LLMs and search technologies in Retrieval Augmented Generation, according to study co-author and Queensland University of Technology Professor Guido Zuccon, Director of AI at the Queensland Digital Health Centre (QDHeC).

“We demonstrate that the interaction between the LLM and the search component is still poorly understood and controllable, resulting in the generation of inaccurate health information,” said Professor Zuccon.

The research’s next steps involve examining how the general population makes use of the health information produced by LLMs.

For more information: Dr. ChatGPT tell me what I want to hear: How different prompts impact health answer correctness, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (2023), DOI: 10.18653/v1/2023.emnlp-main.928