aiqbench analysis
This study reveals that current LLMs prioritize familiarity over suitability, frequently defaulting to Python and overusing libraries like NumPy even when they are suboptimal for the specific task. Such behavior highlights a critical gap in existing evaluations that focus solely on functional correctness, necessitating new benchmarks and fine-tuning strategies that explicitly measure language and library selection fidelity. For practitioners, this underscores the risk of relying on default model outputs for high-performance or specialized projects without manual verification of design choices.
A Study of LLMs' Preferences for Libraries and Programming Languages
arXiv:2503.17181v4 Announce Type: cross Abstract: Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use. To fill this gap, we perform the first empirical study of LLMs' preferences for libraries and programming languages when generating code, covering eight diverse LLMs. We observe a strong tendency to overuse widely adopted libraries such as NumPy; in up to 45% of cases, this usage is not required and deviates from the ground-truth solutions. The LLMs we study also show a significant preference toward Python as their default language. For high-performance project initialisation tasks where Python is not the optimal language, it remains the dominant choice in 58% of cases, and Rust is not used once. These results highlight how LLMs prioritise familiarity and popularity over suitability and task-specific optimality; underscoring the need for targeted fine-tuning, data diversification, and evaluation benchmarks that explicitly measure language and library selection fidelity.
Abstract:Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use. To fill this gap, we perform the first empirical study of LLMs' preferences for libraries and programming languages when generating code, covering eight diverse LLMs. We observe a strong tendency to overuse widely adopted libraries such as NumPy; in up to 45% of cases, this usage is not required and deviates from the ground-truth solutions. The LLMs we study also show a significant preference toward Python as their default language. For high-performance project initialisation tasks where Python is not the optimal language, it remains the dominant choice in 58% of cases, and Rust is not used once. These results highlight how LLMs prioritise familiarity and popularity over suitability and task-specific optimality; underscoring the need for targeted fine-tuning, data diversification, and evaluation benchmarks that explicitly measure language and library selection fidelity.
21 pages, 10 tables, 3 figures. Accepted to Findings of ACL 2026
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
arXiv:2503.17181 [cs.SE]
(or arXiv:2503.17181v4 [cs.SE] for this version)
Full text under the publisher's open licence · 193 words
Related coverage
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
How robust are natural language autoencoders to initialization?
AI Alignment Forum · Jul 10, 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026