aiqbench analysis
This study identifies "layer free-riding" as a specific pathology in Forward-Forward networks where later layers decay exponentially due to margins accumulated by earlier ones, yet demonstrates that fixing this issue yields minimal accuracy gains. The research concludes that architectural choices and dataset characteristics remain far more critical determinants of performance than the specific non-degenerate training rules investigated. Ultimately, while the identified gradient decay is real and repairable, it does not serve as the primary bottleneck limiting model accuracy in standard configurations.
Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant
arXiv:2605.06240v2 Announce Type: replace Abstract: Forward-Forward (FF) training lets each layer learn from a local goodness criterion. In cumulative-goodness variants, later layers can inherit a task that earlier layers have partly separated. We formalize this as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block $d$ decays exponentially with the positive margin accumulated by earlier blocks (Theorem 3.1); a squared-hinge barrier reproduces the pathology. History-free and hardness-gated repairs raise deeper-layer separation by up to $5\times$ on CIFAR-100 and $45\times$ in an 8-block CIFAR-10 model, yet these and a depth-scaled auxiliary term move accuracy by under one percentage point among non-degenerate variants; on Tiny ImageNet, a harder cross-dataset check of the selected configuration, layer health likewise does not track accuracy. A theory-derived attenuation-compensated objective raises deepest-block separation $5.9\times$ at a consistent cost in the FF classifier's own accuracy that a probe trained on the frozen features recovers; in a smaller CIFAR-10 control, most of its separation gain comes from dropping the depth-scaled auxiliary term it replaces. Accuracy invariance is a property of the deployed sum-based scoring rule under redistribution (Proposition 3.2) and, empirically, of the structured trained readouts we test, not of what the network learns: retrained variants disagree on 13-26% of test predictions. Architecture and augmentation matter far more than the non-degenerate training rules studied. Cumulative free-riding is therefore a real and repairable pathology, but for the softplus and squared-hinge objectives, architectures, and datasets we study, it is not the dominant factor limiting accuracy.
Abstract:Forward-Forward (FF) training lets each layer learn from a local goodness criterion. In cumulative-goodness variants, later layers can inherit a task that earlier layers have partly separated. We formalize this as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block $d$ decays exponentially with the positive margin accumulated by earlier blocks (Theorem 3.1); a squared-hinge barrier reproduces the pathology. History-free and hardness-gated repairs raise deeper-layer separation by up to $5\times$ on CIFAR-100 and $45\times$ in an 8-block CIFAR-10 model, yet these and a depth-scaled auxiliary term move accuracy by under one percentage point among non-degenerate variants; on Tiny ImageNet, a harder cross-dataset check of the selected configuration, layer health likewise does not track accuracy. A theory-derived attenuation-compensated objective raises deepest-block separation $5.9\times$ at a consistent cost in the FF classifier's own accuracy that a probe trained on the frozen features recovers; in a smaller CIFAR-10 control, most of its separation gain comes from dropping the depth-scaled auxiliary term it replaces. Accuracy invariance is a property of the deployed sum-based scoring rule under redistribution (Proposition 3.2) and, empirically, of the structured trained readouts we test, not of what the network learns: retrained variants disagree on 13-26% of test predictions. Architecture and augmentation matter far more than the non-degenerate training rules studied. Cumulative free-riding is therefore a real and repairable pathology, but for the softplus and squared-hinge objectives, architectures, and datasets we study, it is not the dominant factor limiting accuracy.
Accepted at NeurIPS 2026 (poster). Camera-ready version: 65 pages (10 pages main text), 24 figures, 45 tables. Code: this https URL ; checkpoints: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.06240 [cs.LG]
(or arXiv:2605.06240v2 [cs.LG] for this version)
Full text under the publisher's open licence · 282 words
