analiza aiqbench
This study introduces HealthBench-Psych, a rigorously validated subset of OpenAI's HealthBench specifically designed to evaluate large language models on mental health queries. By filtering 5,000 conversations down to 610 clinically reviewed cases, the research provides a standardized benchmark to assess model safety, refusal behaviors, and performance consistency across different vendors in psychological support scenarios. The release of this dataset and evaluation pipeline offers developers a critical tool for isolating domain-specific capabilities within general-purpose health benchmarks.
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
arXiv:2608.25071v1 Announce Type: cross Abstract: General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, making domain-specific performance hard to isolate. Mental health is of acute public-health concern as millions of people turn to LLMs for psychological support, and most existing evaluations are bespoke academic benchmarks that are difficult to integrate into developer workflows. We introduce HealthBench-Psych and HealthBench-Psych-Hard. We screened HealthBench's 5,000 physician-rubric conversations for mental-health relevance with a transparent LLM-applied rubric, then validated the subset through two rounds of blinded clinician review with concealed known-exclude controls, yielding 610 conversations (12.2% of the corpus). Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, we find a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges ($\tau \ge 0.92$). We release the subset, pipeline, model responses, grades, and analysis code as a reusable resource.
Abstract:General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, making domain-specific performance hard to isolate. Mental health is of acute public-health concern as millions of people turn to LLMs for psychological support, and most existing evaluations are bespoke academic benchmarks that are difficult to integrate into developer workflows. We introduce HealthBench-Psych and HealthBench-Psych-Hard. We screened HealthBench's 5,000 physician-rubric conversations for mental-health relevance with a transparent LLM-applied rubric, then validated the subset through two rounds of blinded clinician review with concealed known-exclude controls, yielding 610 conversations (12.2% of the corpus). Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, we find a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges ($\tau \ge 0.92$). We release the subset, pipeline, model responses, grades, and analysis code as a reusable resource.
21 pages (6-page main text plus appendices), 2 figures. Code: this https URL Data: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
arXiv:2608.25071 [cs.CL]
(or arXiv:2608.25071v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.25071 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 268 cuvinte
Subiecte conexe
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · 30 iul. 2026
OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
arXiv · 28 iul. 2026
[AINews] GPT-6 Astra: OpenAI’s biggest LLM launch of all time
Latent Space · 4 sept. 2026
Psyche-R1: Towards Reliable Psychological LLMs through Unified Empathy, Expertise, and Reasoning
arXiv · 21 iul. 2026