Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

Abstract:Large language models exhibit remarkable emergent behaviors, yet the physical mechanism governing their collective dynamics remains poorly understood. Cognitive Field Theory predicts that learning organizes collective dynamics through the infrared accumulation of slow relaxation modes, enhancing memory self-energy, long-memory dynamics, and collective susceptibility. Here we test this framework directly in Transformer dynamics. Using publicly available Pythia language models, we extract relaxation spectra from layer Jacobians throughout training, prompt ensembles, network depth, and model scale, allowing the collective observables of Cognitive Field Theory to be measured quantitatively. The measurements reveal pronounced infrared reorganization of the relaxation spectrum. Slow relaxation modes progressively accumulate toward the infrared, producing an approximately flat time-scale density of states, \( \rho(\lambda)\sim\lambda^\beta,\ \beta\simeq-0.1, \) while the corresponding memory kernel exhibits universal scaling, \( K(t)\sim1/t. \) The collective observables further reveal a critical formation process: the memory self-energy reaches a transient maximum during early training before relaxing toward a metastable near-critical regime. Prompt-resolved and token-subspace measurements show that distinct local Jacobians converge toward the same normalized infrared TDOS, consistent with an infrared fixed-point organization under coarse graining. The reproducibility of the same infrared organization across training, prompt ensembles, network depth, and Transformer model scales establishes infrared slow-mode organization as a universal collective principle underlying Transformer dynamics and provides the first quantitative experimental realization of the collective observables introduced by Cognitive Field Theory.

Machine Learning (cs.LG)

arXiv:2607.10923 [cs.LG]

(or arXiv:2607.10923v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2607.10923 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.