aiqbench takeMachine Learning

This research challenges the optimization-centric view of skip connections by proving they fundamentally alter the representational capacity of networks using modern activation functions like SwiGLU and GeGLU. The authors demonstrate that residual-free MLPs and residual blocks are generically disjoint, meaning exact functional recovery via retraining is impossible for frontier models. This finding shifts the discourse from gradient flow benefits to intrinsic architectural limitations in deep learning design.

Sources coveringarXiv, LessWrong, AI Alignment Forum, Fortune

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Can an MLP Absorb Its Own Skip Connection Exactly?

arXivOctober 7, 20261 min read

arXiv:2604.23705v2 Announce Type: replace Abstract: The benefits usually attributed to skip connections are optimization-theoretic: a smoother loss landscape and better gradient propagation. We ask a representational question instead: given a residual block x -> x + MLP(x), does a residual-free MLP of the same width compute the same function? The answer is no, unconditionally and at every depth, for every activation used in current frontier language models (ReLU^2, ReGLU, SwiGLU, GeGLU). For ungated ReLU and GELU absorption is possible, but only on a set of weights of measure zero. The two families are therefore generically disjoint: removing a skip connection and retraining cannot recover exactly the same function at equal width.

Abstract:The benefits usually attributed to skip connections are optimization-theoretic: a smoother loss landscape and better gradient propagation. We ask a representational question instead: given a residual block x -> x + MLP(x), does a residual-free MLP of the same width compute the same function? The answer is no, unconditionally and at every depth, for every activation used in current frontier language models (ReLU^2, ReGLU, SwiGLU, GeGLU). For ungated ReLU and GELU absorption is possible, but only on a set of weights of measure zero. The two families are therefore generically disjoint: removing a skip connection and retraining cannot recover exactly the same function at equal width.

Accepted at the NeurReps Workshop @ NeurIPS 2026 (Extended Abstract Track), nominated for oral presentation. Extended version. OpenReview: this https URL

Machine Learning (cs.LG)

arXiv:2604.23705 [cs.LG]

(or arXiv:2604.23705v2 [cs.LG] for this version)

Full text under the publisher's open licence · 138 words

Read at arXiv ↗Ask about this article