analiza aiqbench
This research exposes a critical flaw in Retrieval-Augmented Generation (RAG) systems, demonstrating that they do not uniformly correct factual errors but instead amplify existing geographic biases in model knowledge. The study reveals that retrieval effectiveness is intrinsically coupled to internal representations, meaning larger models and perfect context alone cannot eliminate these structural disparities without addressing underlying parametric gaps. For developers relying on RAG for enterprise applications, this underscores the necessity of auditing entity representation across diverse regions before deploying systems for high-stakes decision-making.
When RAG Fails to Equalize: Geo-bias in Factual Question Answering over Public Companies
arXiv:2608.25717v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is widely assumed to mitigate factual errors in large language models (LLMs), but it remains unclear whether retrieval uniformly compensates for missing knowledge. We study this question in a controlled factual QA setting over public companies, constructing a benchmark of approximately 2,000 firms across global equity indices. We evaluate six LLMs on four atomic attributes under four conditions: no-context, perfect context, misleading context, and distraction context. We find strong geographic disparities in no-context accuracy, indicating uneven parametric knowledge. While perfect context improves performance, it does not eliminate these gaps: gains are correlated with baseline accuracy, suggesting retrieval effectiveness is coupled to internal representations. Under misleading context, models frequently copy incorrect information. Larger models improve overall performance but do not remove these structural effects. These results challenge the view of RAG as a universal corrective and highlight the interaction between model knowledge, context quality, and entity representation.
Abstract:Retrieval-augmented generation (RAG) is widely assumed to mitigate factual errors in large language models (LLMs), but it remains unclear whether retrieval uniformly compensates for missing knowledge. We study this question in a controlled factual QA setting over public companies, constructing a benchmark of approximately 2,000 firms across global equity indices. We evaluate six LLMs on four atomic attributes under four conditions: no-context, perfect context, misleading context, and distraction context. We find strong geographic disparities in no-context accuracy, indicating uneven parametric knowledge. While perfect context improves performance, it does not eliminate these gaps: gains are correlated with baseline accuracy, suggesting retrieval effectiveness is coupled to internal representations. Under misleading context, models frequently copy incorrect information. Larger models improve overall performance but do not remove these structural effects. These results challenge the view of RAG as a universal corrective and highlight the interaction between model knowledge, context quality, and entity representation.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2608.25717 [cs.CL]
(or arXiv:2608.25717v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.25717 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 247 cuvinte
Subiecte conexe
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
What We Learned Trying to Catch AI Liars: An Aletheia's Quest Retrospective
EleutherAI · 25 aug. 2026
Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values
AI Alignment Forum · 31 iul. 2026
The world's best gradual disempowerment model organism: Frontier AI labs
LessWrong · 29 sept. 2026