analiza aiqbench
LoopFM addresses the diminishing returns of standard knowledge distillation by utilizing historical intermediate embeddings from foundation models to enhance compact vertical models, effectively doubling the knowledge transfer ratio without incurring real-time inference costs. This approach yields significant AUC gains on public benchmarks and translates to measurable conversion improvements in industrial-scale recommendation systems. For practitioners, the method offers a practical pathway to leverage massive model capabilities within constrained serving architectures while maintaining architectural independence.
LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
arXiv:2605.29280v2 Announce Type: replace Abstract: Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose LoopFM (Learning frOm HistOrical RePresentations of FM), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as input features (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong AUC improvements (e.g., 6%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately doubles the knowledge transfer ratio on top of KD, delivering a +0.5% conversion improvement in the first half after its initial launch, and +1.03% and +1.22% conversion improvement from two individual launches in the subsequent half.
Abstract:Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose LoopFM (Learning frOm HistOrical RePresentations of FM), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as input features (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong AUC improvements (e.g., 6%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately doubles the knowledge transfer ratio on top of KD, delivering a +0.5% conversion improvement in the first half after its initial launch, and +1.03% and +1.22% conversion improvement from two individual launches in the subsequent half.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
arXiv:2605.29280 [cs.LG]
(or arXiv:2605.29280v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 196 cuvinte
Subiecte conexe
LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
arXiv · 8 oct. 2026
Modernizing and scaling support operations with generative AI on AWS
AWS · 2 sept. 2026
Building a User Foundation Model for the Open Web
arXiv · 31 iul. 2026
Salesforce after Dreamforce: How the CRM giant can grow beyond its own interface
SiliconANGLE · 19 sept. 2026