Five LLMs show 89.4% to 91.2% accuracy in pharmacy questions

LLM Accuracy in Infectious Disease Education

Large language models—including ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini—demonstrated an overall accuracy rate between 89.4% and 91.2% when answering infectious disease questions from a Doctor of Pharmacy program, Nature reported. The study evaluated 227 exam items, finding that while these models are highly capable of handling standard coursework, they struggle with specific complex tasks like image interpretation and higher-order reasoning.

ChatGPT-4o-mini achieved the highest score among the five models tested, while Claude and Copilot occupied the lower end of the performance spectrum. The research utilized a variety of academic metrics, including Bloom’s taxonomy and categorical topic breakdowns, to determine where these AI tools thrive and where they risk providing misinformation to students and clinicians.

Performance Gaps in Pharmacology and Clinical Reasoning

The study revealed a distinct hierarchy in model proficiency across different subject areas. Pharmacology questions yielded the highest accuracy at 93.6%, followed by therapeutics at 93.0%. Conversely, medicinal chemistry and microbiology presented significant hurdles, with accuracy rates falling to 84% and 87.5%, respectively. These two fields performed significantly worse than therapeutics, indicating that current LLM architectures may lack the specialized depth required for complex molecular and biological classification.

The gap widened further when evaluating the cognitive demand of the questions. Items categorized under “Evaluate/Create” saw an accuracy rate of 85.6%, compared to 92.0% for questions focused on “Remember/Understand.” This suggests that while models are excellent at retrieving established data, they falter when tasked with synthesis or critical clinical application.

The Visual Interpretation Barrier

One of the most striking findings involves the role of visual data in model performance. Questions that included a picture were answered with significantly lower accuracy—just 69%—compared to text-only questions, which saw a 91% success rate. This discrepancy highlights a critical limitation in current digital health tools, as medical education and diagnostic support frequently rely on the analysis of visual diagnostic aids like gram stains, radiographs, or clinical imagery.

Inter-model agreement was moderate, with Cohen’s Kappa Coefficient ranging from 0.497 to 0.613. This variation suggests that different models may arrive at correct answers through different internal logic paths, or conversely, that they share unique blind spots that could lead to consistent errors in a professional setting. The researchers concluded that while these models are promising for educational integration, their current limitations necessitate careful human oversight and rigorous guidance in health-science curricula.