MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA

At our latest Snorkel AI Reading Group, Yuexing Hao, co-founder and COO of MatrAIx and previously a postdoc in MIT’s Healthy ML Group, presented MedPAIR, a dataset that compares which parts of a clinical case physicians and LLMs treat as relevant. Physician trainees and board-certified physicians labeled every sentence in 933 medical exam questions, and six LLMs labeled the same sentences. The question it asks: when a model answers a clinical question correctly, is it reasoning from the same evidence a physician would use? MedPAIR was supported by Snorkel’s Open Benchmarks Grants program. Lightly edited for readability. Moderator: Yuexing Hao is co-founder and COO of MatrAIx, building a new evaluation paradigm for AI agents. Previously, she was a postdoc at MIT’s Healthy ML Group and holds a PhD from Cornell University. Her research focuses on AI for healthcare and human-computer interaction. This work is increasingly important as discussion grows around the future of human work and humanity in the age of AI, particularly in medicine. Snorkel is happy to have supported Yuexing’s work on MedPAIR through the Open Benchmarks Grants program. Please visit us if you would like to learn more. She will also present this work at the upcoming…