Large Language Models Score High on Pharmacy Coursework
Five contemporary large language models scored between 89.4% and 91.2% overall on 227 exam items from an infectious diseases course in a Doctor of Pharmacy program, Nature reported. Researchers evaluated ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini across multiple exam formats, topics, and levels of Bloom’s taxonomy.
ChatGPT-4o-mini achieved the highest overall score among the tested models. Claude and Copilot recorded the lowest overall scores in the comparative evaluation.
Performance Breakdown Across Course Topics
Model accuracy varied significantly depending on the specific subject matter of the exam questions. LLMs scored highest on pharmacology questions at 93.6%, followed by therapeutics at 93.0% and pharmacokinetics/pharmacodynamics at 91.1%.
Microbiology and medicinal chemistry questions yielded lower scores, at 87.5% and 84% respectively. Statistical analysis showed that medicinal chemistry and microbiology performance lagged significantly behind therapeutics, with p-values falling below 0.05.
Models Struggle More with Evaluation Tasks and Image Questions
Questions classified under the Remember/Understand category of Bloom’s taxonomy yielded a 92.0% accuracy rate. Items categorized under Evaluate/Create dropped significantly to 85.6%, marking a statistically significant performance gap.
Exam Data Insight: Questions accompanied by pictures proved particularly difficult for the models. Items with images scored 69% accuracy, compared to 91% for text-only questions.
Inter-model agreement measured via Cohen’s Kappa Coefficient ranged from moderate to substantial, with values spanning from 0.497 to 0.613.
Frequently Asked Questions About AI in Pharmacy Education
Which AI model scored the highest on the infectious diseases exam?
ChatGPT-4o-mini achieved the highest overall accuracy among the five evaluated large language models, scoring at the top of the 89.4% to 91.2% range.
Which topic areas caused the most difficulty for the tested LLMs?
Medicinal chemistry and microbiology recorded the lowest scores at 84% and 87.5% respectively, performing significantly worse than therapeutics questions.
How did visual questions impact model performance?
Questions that included a picture saw a steep drop in accuracy down to 69%, compared to 91% for questions without images.
Worth a look