aiqbenchInformații despre AI & calcul cuantic

...

Română · beta. Traducerea a fost făcută automat și este încă în curs de revizuire. Citește pagina în engleză

perspectiva aiqbenchMachine learning

This research challenges the optimization-centric view of skip connections by proving they fundamentally alter the representational capacity of networks using modern activation functions like SwiGLU and GeGLU. The authors demonstrate that residual-free MLPs and residual blocks are generically disjoint, meaning exact functional recovery via retraining is impossible for frontier models. This finding shifts the discourse from gradient flow benefits to intrinsic architectural limitations in deep learning design.

Surse care acoperăarXiv, LessWrong, AI Alignment Forum, Fortune

Ce este aceastaLectura unui model asupra articolului și a acoperirii conexe, scrisă pentru aiqbench. Nu sunt cuvintele publicației.

Can an MLP Absorb Its Own Skip Connection Exactly?

arXiv7 octombrie 20261 min de cititArticolul și perspectiva sunt în engleză.

arXiv:2604.23705v2 Announce Type: replace Abstract: The benefits usually attributed to skip connections are optimization-theoretic: a smoother loss landscape and better gradient propagation. We ask a representational question instead: given a residual block x -> x + MLP(x), does a residual-free MLP of the same width compute the same function? The answer is no, unconditionally and at every depth, for every activation used in current frontier language models (ReLU^2, ReGLU, SwiGLU, GeGLU). For ungated ReLU and GELU absorption is possible, but only on a set of weights of measure zero. The two families are therefore generically disjoint: removing a skip connection and retraining cannot recover exactly the same function at equal width.

Abstract:The benefits usually attributed to skip connections are optimization-theoretic: a smoother loss landscape and better gradient propagation. We ask a representational question instead: given a residual block x -> x + MLP(x), does a residual-free MLP of the same width compute the same function? The answer is no, unconditionally and at every depth, for every activation used in current frontier language models (ReLU^2, ReGLU, SwiGLU, GeGLU). For ungated ReLU and GELU absorption is possible, but only on a set of weights of measure zero. The two families are therefore generically disjoint: removing a skip connection and retraining cannot recover exactly the same function at equal width.

Accepted at the NeurReps Workshop @ NeurIPS 2026 (Extended Abstract Track), nominated for oral presentation. Extended version. OpenReview: this https URL

Machine Learning (cs.LG)

arXiv:2604.23705 [cs.LG]

(or arXiv:2604.23705v2 [cs.LG] for this version)

Text integral sub licența deschisă a publicației · 138 cuvinte

Citește pe arXiv ↗Întreabă despre acest articol