Towards Effective Theory of LLMs: A Representation Learning Approach

Abstract:We propose Representational Effective Theory (RET), a framework for describing large language model computation in terms of learned macrostates rather than microscopic details. RET learns these macrostates from hidden-state trajectories using a BYOL/JEPA-style self-supervised objective, coarse-graining activations into macrovariables that preserve higher-level structure relevant for prediction and interpretation. We evaluate whether these macrovariables are practically relevant for interpretability: RET yields temporally consistent states that reveal ``mental-state'' trajectories of reasoning, capture high-level semantic structure, support early prediction of behavioral outcomes such as sycophancy and alignment faking, while providing causal handles for steering generations toward interpretable computational phases. Together, these results suggest that LLM computation admits useful effective descriptions via RET: high-level, dynamically meaningful variables for interpretation, prediction, and control.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2605.09294 [cs.LG]

(or arXiv:2605.09294v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.09294 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.