Medical writing with language models: faster but faultier
Part of the Learning with the Machines feature on how UIC educators and researchers are exploring the impact of large language models
For physicians, case reports are an important form of scientific communication. These short summaries, typically of a single patient’s course of disease and treatment, are anecdotal by comparison to large clinical trials, but can offer medical guidance and suggest important research avenues. But physicians are busy people and struggle to find the time to prepare even these short papers.
If large language models have proven any value in their early days, it’s as a time-saver. So a team of College of Medicine physicians wondered: Could this knack for digesting knowledge work for the standardized format of the case report? Dr. Jacqueline Jansz, an internal medicine resident at UI Health, put ChatGPT-as-co-author to the test in a recent publication.
“I thought if this makes case reports easier to write, or even medical literature easier to write, then maybe more people will be open to publishing,” Jansz said. “I think that there’s probably a lot of case reports that go unwritten because of the amount of time it takes to write them.”
Co-authored with former resident Dr. Michael Manansala and Dr. Nadera Sweiss, professor of medicine and chief of rheumatology, the report discusses the treatment of a pregnant woman with the autoimmune disease lupus who suffered from sun-induced facial swelling and rashes. Like most case reports, it provides an overview of the disease, a description of the patient’s diagnosis and treatment, and a discussion of what made the example clinically interesting.
Jansz wrote the details of the patient’s case herself; no private medical information was shared with the language model. But she used ChatGPT to write a first draft of the paper’s introduction and discussion sections, asking it to summarize topics such as photosensitivity in lupus patients and the challenges of using steroid treatment during pregnancy.
The results were fast — but needed a lot of revision, she said. In one case, the model described a rare symptom as common, and it cited imaginary sources — a well-known ChatGPT issue often described as “hallucinations.” In fact, when Sweiss tested the model with another topic in rheumatology, it included a realistic-looking citation listing her as a co-author. But the paper didn’t exist.
“I thought maybe I’m forgetting what I published before,” Sweiss said. “But I searched, and I’ve never published something like this.”
But even with substantial fact-checking and corrections, the paper was written and published faster than any other in her career, Sweiss said. And the physicians see future uses for ChatGPT in medicine, such as preparing individualized information for patients to take home after a clinic visit. Instead of a generic document, physicians could ask the language model to tailor advice to a patient’s demographics, language preferences or educational level.
“It’s going to be a huge benefit to us in terms of time-saving, secretarial work, editing, scheduling patients and explaining things to patients,” Sweiss said. “I think it’s going to be great. I’m very excited about it.”
In a recent article for The Rheumatologist with attorney Peter T. Sadelski, a UIC Law alumnus, Jansz described other potential applications in the clinic, including integration with electronic health record systems to guide physicians in reviewing a patient chart or automatically writing notes during a visit. However, these uses would require further study and legal protections for patient privacy.
“Integration with EHR platforms such as EPIC could revolutionize how we create patient notes,” Jansz said. “Not only would this lead to faster documentation, but it could also help enhance the quality and readability of notes, making them more universally understandable. Such improvements could foster better collaboration and understanding among healthcare professionals, ultimately benefitting patient care.”
