COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal

Title:COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal

Abstract:Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.

28 pages, including Extended Data and Supplementary Information. Code for reproducibility: this https URL

Neurons and Cognition (q-bio.NC); Computation and Language (cs.CL)

arXiv:2610.06615 [q-bio.NC]

(or arXiv:2610.06615v1 [q-bio.NC] for this version)

https://doi.org/10.48550/arXiv.2610.06615 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.