Behavioral Coherence: A Method for Sensitive-Domain LLM Evaluation

Abstract:People use LLMs for reproductive-health questions, including abortion-related support. A response can sound supportive while answers reinforce harmful assumptions: judgment is likely, secrecy is safer, and support is limited. We introduce behavioral coherence evaluation, a design-time method that uses validation evidence from an established instrument to test relations among outputs. Using the Individual Level Abortion Stigma Scale, we prompted five LLMs to complete questionnaires for 627 personas and reviewed flagged patterns with five reproductive-health experts. Models scored personas lower on self-judgment but higher on worries about judgment; most made worries the highest-scoring dimension, although it was lowest in the ILAS reference sample. Four of five models reversed the reference direction by generating significantly higher worries about judgment scores for Black personas. Models defaulted to extreme secrecy after abortion despite varying stigma patterns across personas. Expert review showed that disclosure guidance requires context about relationship safety, legal risk, and trusted support.

Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)

I.2.7; H.5.0; J.3; J.4; K.4.1; K.4.2

arXiv:2512.13142 [cs.AI]

(or arXiv:2512.13142v5 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2512.13142 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.