aiqbench analysis
Crystal introduces a joint ranking method that evaluates all cited papers within a document simultaneously using LLMs, addressing the limitations of isolated citation analysis. By mitigating positional bias through randomized ordering and majority voting, the approach achieves significant accuracy gains and improved efficiency over prior state-of-the-art classifiers. The release of the Crystal-Bank dataset and open-source code provides a scalable resource for more reliable citation impact assessment in research workflows.
Crystal: Characterizing Relative Impact of Scholarly Publications
arXiv:2603.26791v3 Announce Type: replace-cross Abstract: Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose Crystal, which instead jointly ranks all cited papers within a citing paper using large language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references. Crystal outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations. Crystal further gains efficiency through fewer LLM calls and outperforms prior baselines using an open-weight model, enabling scalable, cost-effective citation impact analysis. In a case study of ACL Test-of-Time award-winning papers, we find that Crystal's impact characterizations align closely with long-term scientific recognition. We release Crystal-Bank, a 46.8k-paper dataset with rankings and impact labels, along with code.
Abstract:Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose Crystal, which instead jointly ranks all cited papers within a citing paper using large language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references. Crystal outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations. Crystal further gains efficiency through fewer LLM calls and outperforms prior baselines using an open-weight model, enabling scalable, cost-effective citation impact analysis. In a case study of ACL Test-of-Time award-winning papers, we find that Crystal's impact characterizations align closely with long-term scientific recognition. We release Crystal-Bank, a 46.8k-paper dataset with rankings and impact labels, along with code.
Digital Libraries (cs.DL); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2603.26791 [cs.DL]
(or arXiv:2603.26791v3 [cs.DL] for this version)
Full text under the publisher's open licence · 198 words
Related coverage
Malleability of transformations on the ciphertext in noisy Quantum public key encryption
arXiv · Jul 30, 2026
Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data
arXiv · Jun 3, 2026
Why I Left Google DeepMind
LessWrong · Jul 15, 2026
CFR without Unbiasedness: Deterministic Guarantees for Persistent Public-Chance Schedules
arXiv · Aug 19, 2026