analiza aiqbench
This paper introduces RaLiMPs, a novel method that uses low-degree rational approximations to linearize multilayer perceptrons, offering a more efficient alternative to existing polynomial-based techniques. By constructing kernel representations with narrow bottlenecks, the approach achieves competitive accuracy while significantly reducing inference costs across diverse applications like implicit 3D modeling and large language models. The work provides theoretical guarantees for these approximations, establishing a favorable trade-off between model efficiency and performance.
Rationally Linearized MLPs
arXiv:2610.13568v1 Announce Type: new Abstract: We present Rationally Linearized Multilayer Perceptrons(RaLiMPs), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy--efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.
Abstract:We present Rationally Linearized Multilayer Perceptrons(RaLiMPs), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy--efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.
Preprint. Comments welcome
Machine Learning (cs.LG)
arXiv:2610.13568 [cs.LG]
(or arXiv:2610.13568v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.13568 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 193 cuvinte
Subiecte conexe
The Entangled Dimensions of Decision Theory
LessWrong · 30 iul. 2026
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · 23 iul. 2026
X-OPM: Explainable Automatic Digital On-Chip Power Modeling for Enhanced Robustness
arXiv · 7 oct. 2026
Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
arXiv · 30 sept. 2026