analiza aiqbench
This study demonstrates that for social highlighting tasks, scaling a model specifically on reader-highlight data yields significantly better precision than simply leveraging larger frontier language models as zero-shot teachers or rankers. The findings suggest that domain-specific training on interaction data is more critical than raw parameter count or generalist model strength when predicting human convergence points in text.
More Reader Data or a Better Teacher? Updating a Model of Crowd Highlights in Late 2026
arXiv:2610.17768v1 Announce Type: cross Abstract: A model trained on readers' highlights can show a newcomer where earlier readers converged on a page, even on pages with few readers. A year of stronger language models suggests two upgrades: use them directly, or use them as teachers. We compare both with a third, more reader data, on one social highlighting platform, with the comparisons, the selection rules and a sealed set of fresh pages fixed in advance. On 215 development pages, seven language models released by October 2026 rank sentences zero-shot at an average precision of at most 0.3744 against the sentences half the readers marked most, and 0.3931 when fused with reading position, while a 3-billion-parameter model trained on reader highlights alone reaches 0.4346. Used as the teacher that breaks ties among readers' counts, the stronger fusion changes a trained model by +0.0013 [-0.0012, +0.0040], whereas going from half to all of its training pages adds +0.0242 [+0.0179, +0.0301]. A 27-billion-parameter backbone on the reader-only recipe adds +0.0225 on the development pages; on 131 fresh pages, read once and too few for a confirmatory comparison, its difference from the 3-billion model is unresolved, both score above the fusion, and in an exploratory split their lead over the previous model sits on pages from domains that their fresh training pages also cover. In this recipe, more reader data improves a trained model where a stronger tie-breaking teacher yields no material gain, and frontier models used as rankers still trail a small model trained on readers.
Abstract:A model trained on readers' highlights can show a newcomer where earlier readers converged on a page, even on pages with few readers. A year of stronger language models suggests two upgrades: use them directly, or use them as teachers. We compare both with a third, more reader data, on one social highlighting platform, with the comparisons, the selection rules and a sealed set of fresh pages fixed in advance. On 215 development pages, seven language models released by October 2026 rank sentences zero-shot at an average precision of at most 0.3744 against the sentences half the readers marked most, and 0.3931 when fused with reading position, while a 3-billion-parameter model trained on reader highlights alone reaches 0.4346. Used as the teacher that breaks ties among readers' counts, the stronger fusion changes a trained model by +0.0013 [-0.0012, +0.0040], whereas going from half to all of its training pages adds +0.0242 [+0.0179, +0.0301]. A 27-billion-parameter backbone on the reader-only recipe adds +0.0225 on the development pages; on 131 fresh pages, read once and too few for a confirmatory comparison, its difference from the 3-billion model is unresolved, both score above the fusion, and in an exploratory split their lead over the previous model sits on pages from domains that their fresh training pages also cover. In this recipe, more reader data improves a trained model where a stronger tie-breaking teacher yields no material gain, and frontier models used as rankers still trail a small model trained on readers.
10 pages, 4 tables, 1 figure. Ancillary files include the pre-registration with its amendments, the results write-up, the aggregate artifacts every reported number is generated from, and the prompt, scoring, target and contrast code
Information Retrieval (cs.IR); Computation and Language (cs.CL)
arXiv:2610.17768 [cs.IR]
(or arXiv:2610.17768v1 [cs.IR] for this version)
https://doi.org/10.48550/arXiv.2610.17768 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 304 cuvinte
Subiecte conexe
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers
Hugging Face · 26 aug. 2026
Research Lessons I Learned at MATS & Redwood
LessWrong · 12 oct. 2026