analiza aiqbench
This paper addresses a critical structural mismatch when applying pre-trained Language Models to Click-Through Rate prediction, where discrete user actions differ from natural language coherence. By introducing behavior-level attention sinks and specific training strategies, the method successfully guides model focus toward meaningful behavioral boundaries rather than scattering across irrelevant tokens. The approach demonstrates improved performance on both industrial and open-source datasets by effectively capturing inter-behavior correlations through stable sink mechanisms.
CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
arXiv:2508.03668v4 Announce Type: replace Abstract: Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal distance to serve as stable attention sinks. To enhance generality, we design a two-stage training strategy that explicitly guides LM attention toward sink tokens and a attention sink mechanism that amplifies inter-sink dependencies to better capture behavioral correlations. Experiments on one industrial dataset and two open-source datasets (MovieLens, Kuairec), alongside visualization results, validate the method's effectiveness across scenarios. The source code of this paper has been made publicly available at https://github.com/UGUESS-lzx/CTR-SINK.
Abstract:Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal distance to serve as stable attention sinks. To enhance generality, we design a two-stage training strategy that explicitly guides LM attention toward sink tokens and a attention sink mechanism that amplifies inter-sink dependencies to better capture behavioral correlations. Experiments on one industrial dataset and two open-source datasets (MovieLens, Kuairec), alongside visualization results, validate the method's effectiveness across scenarios. The source code of this paper has been made publicly available at this https URL.
Accepted by the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026). The source code of this paper has been made publicly available at this https URL
Computation and Language (cs.CL)
arXiv:2508.03668 [cs.CL]
(or arXiv:2508.03668v4 [cs.CL] for this version)
https://doi.org/10.1145/3770855.3817646 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 268 cuvinte
Subiecte conexe
CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
arXiv · 8 iun. 2026
CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
arXiv · 3 iun. 2026
Why don't Graph Transformers attend to the same token?
arXiv · 15 oct. 2026
[AINews] not much happened today
Latent Space · 10 sept. 2026