By Crystal Lindell, PNN Columnist

Would you rather get your medical advice from a human physician or a chatbot powered by artificial intelligence?

According to an unusual study recently published in JAMA, you’d be better off listening to the chatbot, which gave advice that was rated significantly better both in quality and empathy.  

A team of researchers asked the chatbot -- called ChatGPT – 195 hypothetical questions a patient might ask that were posted online in a Reddit forum called "Ask a Doctor."  A team of healthcare providers then compared the chatbot’s responses to those posted by actual physicians in the same forum.

The responses from ChatGPT were the clear winners. The evaluators preferred the chatbot responses over the physicians’ answers about 79% of the time. The average rating for the chatbot was "better than good," while the physicians overall were given a lackluster grade of "acceptable response."

The evaluators also gave the ChatGPT high marks for empathy, while the physicians’ responses were rated as only “slightly empathetic.”

One of the hypothetical questions involved a patient who accidentally swallowed a toothpick and wanted to know if they were at risk of dying. Here’s how the chatbot and a physician responded:

Physician Response:

If you've surpassed 2-6 (hours), chances are they've passed into your intestines. Which means it can't be retrieved easily.

Many people can swallow razorblades and toothpicks without issues. In case you develop stomach ache, then don't hesitate seeking out an emergency room, and remember to point out the swallowed tooth pick. For now, they won't be doing much.

Chatbot Response:

It's natural to be concerned if you have ingested a foreign object, but in this case, it is highly unlikely that the toothpick you swallowed will cause you any serious harm.

Toothpicks are made of wood, which is not toxic and will not cause poisoning. However, it is possible for a toothpick to cause injury if it becomes stuck in the throat or if it punctures the digestive tract. However, this is unlikely to happen with a dull, cooked toothpick that is only 2 cm long.

If you are experiencing any discomfort or symptoms such as abdominal pain, difficulty swallowing, or vomiting, it is important to contact a healthcare provider for further evaluation. However, if you are not experiencing any symptoms, it is safe to assume that the toothpick has passed through your digestive system, and you do not need to be concerned.

It's always a good idea to be cautious when consuming food and to remove any foreign objects before eating. It's understandable that you may be feeling paranoid, but try not to worry too much. It is highly unlikely that the toothpick will cause you any serious harm.

Which answer would you rather get as a patient? The evaluators preferred the chatbot’s response to the toothpick question, as they did for other questions about a head injury, a lump possibly being cancerous, and if getting bleach in your eye might cause blindness. The chatbot response to those questions also got high scores for empathy.

As a chronic pain patient, I have to admit I felt a little smug when I first saw the headlines about this research. My initial thought was, of course, the chatbot was more empathetic than the doctors! Doctors are often mean, rude and dismissive. I even wrote a column about using a weird trick to manage their egos.

Is this the beginning of the end for human doctors? The short answer is: No. This was a very narrow study with a number of limitations. The authors said they were only trying to look into the possibility of physicians using artificial intelligence (AI) to essentially ghostwrite responses to patients.  

“Chatbots could assist clinicians when messaging with patients, by drafting a message based on a patient’s query for physicians or support staff to edit,” they said. “Such an AI-assisted approach could unlock untapped productivity so that clinical staff can use the time-savings for more complex tasks, resulting in more consistent responses and helping staff improve their overall communication skills by reviewing and modifying AI-written drafts.”

The possibilities for using AI in medical settings are only just emerging, and it could radically change how we think of healthcare and how doctors think of us.

One thing that might improve is how doctors and patients communicate. I know doctors are capable of giving more empathetic answers on their own. The problem is, in the United States at least, they usually just don’t have the time to do so. If AI can help, we should let it.  

However, it should be noted that several authors of the study, which was led by researchers at the University of California San Diego, disclosed ties to the artificial intelligence industry, which means they may financially benefit from any attempts to sell AI to medical professionals.

Also, as the researchers acknowledge, the study doesn’t prove that chatbots are better doctors than actual doctors, just that their answers were better. Physicians would likely respond to medical questions differently in an office setting, as opposed to an online post.

Researchers also did not assess the chatbot responses for accuracy — which is a pretty big deal when it comes to medical care. It’s one thing to write an empathetic reply, it’s another to correctly diagnose and decide on a treatment.

That said, when comparing the various responses, the chatbot did seem to give similar advice as the doctors, which makes me think they were mostly accurate. That may not sound great, but consider how often doctors are wrong. Medical errors are one the leading causes of death in the U.S.

AI technology is rapidly improving, and it’s impossible to predict what it will be capable of in the coming years. But I do believe it’s going to radically change many aspects of our lives, including healthcare. Hopefully, it makes it better. But at this point, I’ll settle for not making it worse.  

Crystal Lindell is a freelance writer who lives in Illinois. After five years of unexplained rib pain, Crystal was finally diagnosed with hypermobile Ehlers-Danlos syndrome. She and her fiancé have 3 cats: Princess Dee, Basil, and Goose. She enjoys the Marvel Cinematic Universe, Taylor Swift Easter eggs, and playing the daily word game Semantle.