A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

arXiv:2608.14689v1 Announce Type: new Abstract: Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently. This paper presents a reproducibility study of partial residual ablations in Pre-LN GPT-style transformers trained at two scales (10M and 124M parameters). I compare four architectural configurations by selectively removing the attention residual connection, the feed-forward residual connection, or both. Across all experiments, removing the attention residual (FFNOnly) consistently causes deterministic collapse to the No-Residual performance floor. In contrast, removing the feed-forward residual (AttnOnly) exhibits a reproducible recovery effect at 10M scale under a controlled 8-seed deterministic study, while its behavior at 124M remains unresolved because of substantial seed variance. During the investigation, I identified and corrected an experimental measurement confound in runtime gain scaling and document both the failed intermediate reproduction and the subsequent controlled replication. Based on the empirical results, I propose a cross-position routing hypothesis to explain the observed asymmetry while explicitly distinguishing confirmed findings from unresolved questions. To support reproducibility, I release the complete source code, experiment configurations, checkpoints, training logs, and all experimental results, including intermediate non-reproducing runs.

Abstract:Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently. This paper presents a reproducibility study of partial residual ablations in Pre-LN GPT-style transformers trained at two scales (10M and 124M parameters). I compare four architectural configurations by selectively removing the attention residual connection, the feed-forward residual connection, or both. Across all experiments, removing the attention residual (FFNOnly) consistently causes deterministic collapse to the No-Residual performance floor. In contrast, removing the feed-forward residual (AttnOnly) exhibits a reproducible recovery effect at 10M scale under a controlled 8-seed deterministic study, while its behavior at 124M remains unresolved because of substantial seed variance. During the investigation, I identified and corrected an experimental measurement confound in runtime gain scaling and document both the failed intermediate reproduction and the subsequent controlled replication. Based on the empirical results, I propose a cross-position routing hypothesis to explain the observed asymmetry while explicitly distinguishing confirmed findings from unresolved questions. To support reproducibility, I release the complete source code, experiment configurations, checkpoints, training logs, and all experimental results, including intermediate non-reproducing runs.

Machine Learning (cs.LG)

arXiv:2608.14689 [cs.LG]

(or arXiv:2608.14689v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2608.14689 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

Subiecte conexe