Decades-Old DXplain Edges Out ChatGPT and Gemini in Diagnostic Test, Study Finds

In 1984, computer scientists at Massachusetts General Hospital built a program called DXplain. It drew on thousands of disease profiles, clinical findings and data points to generate a ranked list of possible diagnoses for clinicians. Four decades later, researchers at the hospital's Laboratory of Computer Science put it up against ChatGPT and Gemini, the large language models now drawing so much attention in medicine.

DXplain is a diagnostic decision support system, or DDSS, a category of AI that medical professionals have used for decades to streamline diagnosis. MGH is a founding member of the Mass General Brigham healthcare system, and its own system has evolved over those years. The question for the investigators was how the old tool compared with the newer generative models.

The team used 36 patient cases spanning racial, ethnic, age and gender categories. For each case, each of the three systems was asked to suggest possible diagnoses twice: once with laboratory data and once without it.

With lab data, all three listed the correct diagnosis most of the time. DXplain did so in 72 percent of cases, ChatGPT in 64 percent and Gemini in 58 percent.

Without lab data, the figures dropped. DXplain still listed the correct diagnosis 56 percent of the time, ahead of ChatGPT at 42 percent and Gemini at 39 percent. The researchers note that these no-lab differences were not statistically significant. The overall finding is that DXplain performed somewhat better, while the language models also did well.

The researchers also saw that the DDSS and the language models each caught certain diseases the others missed. That overlap, they say, suggests the approaches could work together. Their results appear in JAMA Network Open.

Co-author Edward Hoffer, MD, of the Laboratory of Computer Science at MGH, said the excitement over language models makes it easy to forget that the first AI systems used successfully in medicine were expert systems like DXplain. Corresponding author Mitchell Feldman, MD, also of the lab, said such systems can expand clinicians' diagnoses by recalling information a physician may forget in the heat of the moment, and without the common flaws of human reasoning. He said combining the explanatory strengths of existing diagnostic systems with the linguistic abilities of language models should enable better automated decision support and better patient outcomes.

Preliminary work building on the findings points to one way of doing that. Language models could pull clinical findings out of narrative text, and those findings could then be fed into a DDSS. The researchers expect that would improve both systems and their diagnostic conclusions. That hybrid approach is still at an early stage.