aiqbench analysis
StreamProfileBench introduces a critical new benchmark for evaluating how Large Language Models handle the continuous evolution of user interests in real-time streaming environments. The study highlights that current models suffer from conservative biases, failing to accurately track interest decay as new content arrives dynamically. This work is essential for advancing personalized systems beyond static snapshots by establishing rigorous standards for temporal state maintenance in AI applications.
StreamProfileBench: A Benchmark for Fine-Grained User Profile Inference in Real-World Streaming Scenarios
arXiv:2605.25758v3 Announce Type: replace Abstract: Large Language Models (LLMs) have reshaped user profiling, yet current evaluations mainly focus on static data snapshots. This paradigm overlooks the reality of personalized systems, where User-Generated Content (UGC) arrives continuously and fine-grained profiles evolve rapidly. To bridge this gap, we introduce StreamProfileBench, a large-scale benchmark for fine-grained streaming user profiling. We formalize streaming user profiling as a continuous state maintenance task and curate a highly authentic dataset comprising over 120,000 UGC posts from 7,000+ real users across five diverse platforms. By leveraging the temporal correlation of user interests, we further propose a novel, annotation-free evaluation framework. Extensive experiments across 14 leading LLMs reveal that continuous profile updating remains an open challenge. Models exhibit a systemic conservative bias, over-retaining past interests while failing to recognize interest decay. Ablation experiments further validate the practical utility and necessity of the streaming paradigm. Data and code are hosted in https://github.com/WaterWang-001/StreamProfileBench.
Abstract:Large Language Models (LLMs) have reshaped user profiling, yet current evaluations mainly focus on static data snapshots. This paradigm overlooks the reality of personalized systems, where User-Generated Content (UGC) arrives continuously and fine-grained profiles evolve rapidly. To bridge this gap, we introduce StreamProfileBench, a large-scale benchmark for fine-grained streaming user profiling. We formalize streaming user profiling as a continuous state maintenance task and curate a highly authentic dataset comprising over 120,000 UGC posts from 7,000+ real users across five diverse platforms. By leveraging the temporal correlation of user interests, we further propose a novel, annotation-free evaluation framework. Extensive experiments across 14 leading LLMs reveal that continuous profile updating remains an open challenge. Models exhibit a systemic conservative bias, over-retaining past interests while failing to recognize interest decay. Ablation experiments further validate the practical utility and necessity of the streaming paradigm. Data and code are hosted in this https URL.
Computation and Language (cs.CL)
arXiv:2605.25758 [cs.CL]
(or arXiv:2605.25758v3 [cs.CL] for this version)
Full text under the publisher's open licence · 161 words
Related coverage
ROCm 10.0: A Decade of Open Compute, Built for the Age of Agentic AI
AMD · Aug 27, 2026
Nemotron 3.5 Content Safety: Customizable Multimodal Safety for Global Enterprise AI
Hugging Face · Jun 4, 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Launching UI for generative AI inference recommendations in Amazon SageMaker AI
AWS · Jul 13, 2026