Eka’s Weekly Roundup (19 May 2023)

Issue 32 l GPT vs. Clinicians

May 19, 2023

The University of California San Diego published a paper showing Chat GPT’s performance against doctors when giving medical advice. As reported widely (CNN, Guardian, Daily Mail), Chat GPT showed better scores on patient empathy compared to physician responses when replying to simple medical questions. While this is very early days for large language models (LLMs) in healthcare, this could become a tool for clinicians to reduce workloads, improve patient outcomes, and create efficiencies in a strained healthcare system.

The AI will see you now 🗞️

Comparing medical responses to social media posts…

In late April of this year, academics from UC San Diego compared the quality of responses from physicians and AI chatbots to medical questions. The study evaluated the effectiveness of ChatGPT in providing quality and empathetic responses to patient questions. The researchers used a public database of questions from Reddit's r/AskDocs forum. They randomly selected 195 question-answer exchanges where a verified physician had responded. Chatbot responses were generated by inputting the original question into a fresh session without prior questions asked. A team of licensed healthcare professionals evaluated the responses.

… with one example of how GPT can outperform physicians.

In one example highlighted in the study, a patient asked a question on a social media forum around the risk of losing vision after getting bleach in the eye. ChatGPT began its response by expressing apologies for causing concern. Then, it provided seven additional sentences offering advice and encouragement, reassuring the patient about the low likelihood of going blind in such a situation. On the other hand, a physician responded with a brief comment stating, "sounds like you will be fine," along with the phone number for Poison Control.

The results showed positive support for GPT…

The results showed that the evaluators preferred chatbot responses over physician responses in 78.6% of the evaluations. Chatbot responses were rated higher in quality and empathy compared to physician responses. The chatbot responses were longer in length and provided more detailed information. The prevalence of good or very good quality responses was 3.6 times higher for the chatbot compared to physicians, and the prevalence of empathetic or very empathetic responses was 9.8 times higher for the chatbot.

… but the road ahead will require further research and regulation.

The study suggests that AI chatbot assistants have the potential to provide quality and empathetic responses to patient questions in an online forum. As The Guardian reported, Prof James Davenport form the University of Bath commented: “The paper does not say that ChatGPT can replace doctors, but does, quite legitimately, call for further research into whether and how ChatGPT can assist physicians in response generation.” There are well documented risks of GPT hallucinating (Alkaissi and McFarlane, 2023) (Beutel Geerits and Kielstein, 2023) (Zapier).

Further research is needed to explore the use of chatbots in clinical settings, such as drafting responses for physicians to review. Randomized trials could assess whether using AI assistants can improve responses, reduce clinician burnout, and enhance patient outcomes.