analiza aiqbench
CTR-Sink addresses the structural mismatch between discrete user behavior sequences and continuous natural language pre-training by introducing behavior-level attention sinks. This framework dynamically regulates attention aggregation using recommendation-specific signals like temporal distance to prevent semantic fragmentation and improve focus on meaningful interactions. The proposed two-stage training strategy effectively guides language models to capture inter-behavior correlations, validating its effectiveness across industrial and open-source datasets.
CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
arXiv:2508.03668v2 Announce Type: replace Abstract: Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal distance to serve as stable attention sinks. To enhance generality, we design a two-stage training strategy that explicitly guides LM attention toward sink tokens and a attention sink mechanism that amplifies inter-sink dependencies to better capture behavioral correlations. Experiments on one industrial dataset and two open-source datasets (MovieLens, Kuairec), alongside visualization results, validate the method's effectiveness across scenarios.
Abstract:Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal distance to serve as stable attention sinks. To enhance generality, we design a two-stage training strategy that explicitly guides LM attention toward sink tokens and a attention sink mechanism that amplifies inter-sink dependencies to better capture behavioral correlations. Experiments on one industrial dataset and two open-source datasets (MovieLens, Kuairec), alongside visualization results, validate the method's effectiveness across scenarios.
Computation and Language (cs.CL)
arXiv:2508.03668 [cs.CL]
(or arXiv:2508.03668v2 [cs.CL] for this version)
https://doi.org/10.1145/3770855.3817646 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 223 cuvinte
Subiecte conexe
CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
arXiv · 4 aug. 2026
CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
arXiv · 8 iun. 2026
Why don't Graph Transformers attend to the same token?
arXiv · 15 oct. 2026
[AINews] not much happened today
Latent Space · 10 sept. 2026