aiqbench analysis
The paper introduces FAM-Bench, a benchmark of 2,500 questions that tests whether vision-language models can judge whether a dish suits a person with one of 13 diet-related health conditions, and whether those judgments are backed by food evidence. Its main finding is that models often reach suitability decisions their stated rationales don't support, and they perform poorly at ranking dishes comparatively. For readers, the result is a useful caution that recognizing ingredients or nutrients doesn't establish reliable, grounded health reasoning, and that rationale quality needs separate evaluation.
FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models
arXiv:2605.31410v2 Announce Type: replace Abstract: Food AI can recognize dishes, ingredients, and nutrients. However, these abilities do not show whether a dish is suitable for a person with a specific health condition. They also do not test whether such decisions are grounded in the relevant food evidence. To address this gap, we introduce FAM-Bench, a Food-as-Medicine benchmark comprising 2,500 question instances constructed from 1,811 recipes across 13 diet-related health conditions. Gold annotations are derived from rule-based dietary guidance informed by published sources and nutrition professionals. The benchmark contains two complementary tasks: dish-level suitability assessment and comparative dish analysis. We evaluate open-weight and proprietary vision-language models under four prompting modes. Results reveal a substantial gap between suitability decisions and their supporting rationales, as well as limited performance on comparative ranking. These findings highlight persistent challenges in grounded, condition-aware food reasoning.
Abstract:Food AI can recognize dishes, ingredients, and nutrients. However, these abilities do not show whether a dish is suitable for a person with a specific health condition. They also do not test whether such decisions are grounded in the relevant food evidence. To address this gap, we introduce FAM-Bench, a Food-as-Medicine benchmark comprising 2,500 question instances constructed from 1,811 recipes across 13 diet-related health conditions. Gold annotations are derived from rule-based dietary guidance informed by published sources and nutrition professionals. The benchmark contains two complementary tasks: dish-level suitability assessment and comparative dish analysis. We evaluate open-weight and proprietary vision-language models under four prompting modes. Results reveal a substantial gap between suitability decisions and their supporting rationales, as well as limited performance on comparative ranking. These findings highlight persistent challenges in grounded, condition-aware food reasoning.
Artificial Intelligence (cs.AI)
arXiv:2605.31410 [cs.AI]
(or arXiv:2605.31410v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.31410 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 226 words