Repository
MedXpertQA
[](https://arxiv.org/abs/2501.18362) [.
</details>
✨Features
- Next-Generation Multimodal Medical Evaluation: MedXpertQA MM introduces expert-level medical exam questions with diverse images and rich clinical information, including patient records and examination results, setting it apart from traditional medical multimodal benchmarks with simple QA pairs generated from image captions.
- Highly Challenging: MedXpertQA introduces high-difficulty medical exam questions and applies rigorous filtering and augmentation, effectively addressing the insufficient difficulty of existing benchmarks like MedQA. The Text and MM subsets are currently the most challenging benchmarks in their respective fields.
- Clinical Relevance: MedXpertQA incorporates specialty board questions to improve clinical relevance and comprehensiveness by collecting questions corresponding to 17/25 member board exams (specialties) of the American Board of Medical Specialties. It showcases remarkable diversity across multiple dimensions.
- Mitigating Data Leakage: We perform data synthesis to mitigate data leakage risk and conduct multiple rounds of expert reviews to ensure accuracy and reliability.
- Reasoning-Oriented Evaluation: Medicine provides a rich and representative setting for assessing reasoning abilities beyond mathematics and code. We develop a reasoning-oriented subset to facilitate the assessment of o1-like models.
📊Leaderboard
We evaluate 17 leading proprietary and open-source LMMs and LLMs including advanced inference-time scaled models with a focus on the latest progress in medical reasoning capabilities.
Further details are available in the leaderboard and the paper.
🔧Usage
- Clone the Repository:
git clone https://github.com/TsinghuaC3I/MedXpertQA
cd MedXpertQA/eval- Install Dependencies:
pip3 install -r requirements.txt- Inference:
bash scripts/run.shThe run.sh script performs inference by calling main.py, which offers additional features such as multithreading. Additionally, you can modify model/api_agent.py to support more models.
- Evaluation:
We provide a script eval.ipynb to calculate accuracy on each subset.
[!NOTE]
Please use this script when evaluating the QVQ and DeepSeek-R1. Through case studies, we found that the answer cleaning function in the utils.py is unsuitable for these two models.
📨Contact
-
Shang Qu: [email protected]
-
Ning Ding: [email protected]
⚖️License
This project is licensed under the MIT License.
🎈Citation
If you find our work helpful, please use the following citation.
@article{zuo2025medxpertqa,
title={Medxpertqa: Benchmarking expert-level medical reasoning and understanding},
author={Zuo, Yuxin and Qu, Shang and Li, Yifei and Chen, Zhangren and Zhu, Xuekai and Hua, Ermo and Zhang, Kaiyan and Ding, Ning and Zhou, Bowen},
journal={arXiv preprint arXiv:2501.18362},
year={2025}
}