ReBeCA: Unveiling Interpretable Behavior Hierarchy behind the Iterative Self-Reflection of Language Models with Causal Analysis

Abstract:While self-reflection can enhance language model reliability, its underlying mechanisms remain opaque, with existing analyses often yielding correlation-based insights that fail to generalize. To address this, we introduce **ReBeCA** (self-**Re**flection **Be**havior explained through **C**ausal **A**nalysis), a framework for analyzing the interpretable behavioral hierarchy governing the self-reflection outcome. By modeling self-reflection trajectories as causal graphs, ReBeCA selects observed parent candidates and evaluates their stability through a three-stage ICP-based pipeline. In a controlled Qwen3 case study, we establish three critical findings: (1) Behavioral hierarchy: Semantic behaviors of the model influence final self-reflection results hierarchically: directly or indirectly; (2) Causation matters: Generalizability in self-reflection effects is limited to just a few semantic behaviors; (3) More $\neq$ better: The confluence of seemingly positive semantic behaviors, even among direct causal factors, yield no additive gain. ICP-based verification identifies sparse causal parents achieving up to $49.6\%$ structural likelihood gains relative to the dense full-set association baseline across the studied task subsets. A controlled prompt-based behavioral intervention on a novel dataset provides out-of-distribution validation ($p = .013, \eta^2_\mathrm{p} = .071$). The present study focuses on the Qwen3 family under fixed-round Self-Refine.

Computation and Language (cs.CL)

arXiv:2602.06373 [cs.CL]

(or arXiv:2602.06373v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2602.06373 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.