A4-a - A Multi-Agent Framework for Enhanced Diagnostic Accuracy: Achieving State-of-the-Art Performance with Smaller Models

Saturday, October 11, 2025

10:00 AM - 12:00 PM PDT

Location: Exhibit Hall F, G & H

Presenting Author(s)

TL

Troy Liu

Student
Green Valley High School, United States

Co-Author(s)

Shengjie Zhai

Assistant Professor
University of Nevada Las Vegas
Las Vegas, Nevada, United States
MZ

Ming Zhu

Assitant Professor in Residence
University of Nevada Las Vegas
Las Vegas, Nevada, United States

Introduction: : While large language models (LLMs) show significant promise in medical diagnostics, single-agent architectures often lack the depth of reasoning, consensus-building, and domain specialization essential for accurate clinical decision-making. Multi-agent frameworks based on the Model Context Protocol (MCP) enable specialized agents to independently assess patient information and collaborate to reach a consensus diagnosis. This study evaluates a six-agent MCP-based AI system designed to emulate the reasoning process of a team of medical specialists, using the MedMCQA dataset as a surrogate diagnostic benchmark, and examines how performance scales with dataset size.

Materials and

Methods: : The system utilized a six-agent MCP framework, with each agent built upon the GPT-4o-mini base model. Five "virtual specialist" agents—representing Physiology, Pathology, Clinical Medicine, Treatment Planning, and Public Health—independently analyzed a shared clinical context file to generate reasoned diagnostic conclusions. A sixth "chief physician" agent then synthesized the specialist inputs through a majority-voting mechanism to produce a final consensus diagnosis. The framework's accuracy was first evaluated against a single-agent GPT-4o-mini baseline on the MedMCQA validation set, which contains approximately 4,183 cases. To assess performance scalability, the model was also tested using the MedQA-USMLE training split of approximately 11,000 cases in the shared clinical context file.

Results, Conclusions, and Discussions:: The multi-agent framework demonstrated a significant performance improvement over the single-agent baseline, which achieved 73.0% accuracy on the MedMCQA dataset. Each of the five virtual specialists surpassed this baseline, with individual accuracies ranging from 76.2% to 76.6%. The final consensus diagnosis reached an accuracy of 76.5%, a performance gain of over 3%. When tested on the larger MedQA-USMLE dataset, the consensus accuracy saw a slight increase to 76.7%. This suggests that collaborative architecture provides considerable benefits in limited-data scenarios, though these advantages may plateau as dataset size increases.
Critically, our framework's performance of 76.5%-76.7% matches or exceeds the highest reported score on the MedMCQA benchmark (76.6%), which was achieved by a much larger Llama-3.1-70B model. This result demonstrates that a sophisticated multi-agent architecture can achieve state-of-the-art diagnostic accuracy while operating with significantly lower computational overhead than larger, monolithic models.

Acknowledgements and/or References (Optional): : Troy Liu1, Ming Zhu2, Shengjie(Patrick) Zhai2*
2. Electrical and Computer Engineering Department,
University of Nevada Las Vegas
4505 S. Maryland PKWY, Las Vegas, NV 89154
*Corresponding author