analiza aiqbench
This arXiv paper is primarily about how LLMs respond to common French input variations, such as dropped apostrophes, missing diacritics, and keyboard typos. It builds a 12,600-instance benchmark that measures both accuracy and prediction reliability (Brier score, NLL), and finds that accuracy degrades with perturbation intensity while calibration and input restoration help. For practitioners, the useful takeaway is that scale does not reliably fix confidence quality under noisy input, so calibration and robustness checks should be evaluated separately from accuracy.
Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants
arXiv:2610.33493v1 Announce Type: new Abstract: French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain insufficiently understood. To address this gap, we construct a controlled French surface-variant benchmark covering seven task types, with 4,200 original-low-high triplets (12,600 input instances). Each high-intensity variant retains all low-intensity edits and adds further changes. We evaluate six models from two LLM families. The results show that accuracy decreases and Brier score and negative log-likelihood (NLL) increase with perturbation intensity. Larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently decrease with scale. Keyboard typos cause the largest losses in accuracy and the quality of predicted probabilities at the predefined high-intensity settings. Finally, calibration temperatures fitted on original inputs transfer effectively to surface variants. Combining calibration with input restoration further improves Brier score and NLL over either method alone while preserving restored accuracy. Our benchmark provides a unified framework for jointly evaluating task accuracy and prediction reliability under common French surface variations.
Abstract:French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain insufficiently understood. To address this gap, we construct a controlled French surface-variant benchmark covering seven task types, with 4,200 original-low-high triplets (12,600 input instances). Each high-intensity variant retains all low-intensity edits and adds further changes. We evaluate six models from two LLM families. The results show that accuracy decreases and Brier score and negative log-likelihood (NLL) increase with perturbation intensity. Larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently decrease with scale. Keyboard typos cause the largest losses in accuracy and the quality of predicted probabilities at the predefined high-intensity settings. Finally, calibration temperatures fitted on original inputs transfer effectively to surface variants. Combining calibration with input restoration further improves Brier score and NLL over either method alone while preserving restored accuracy. Our benchmark provides a unified framework for jointly evaluating task accuracy and prediction reliability under common French surface variations.
Computation and Language (cs.CL)
arXiv:2610.33493 [cs.CL]
(or arXiv:2610.33493v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.33493 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 274 cuvinte
Subiecte conexe
Are Benchmarks Reliable? Toward Structural Diagnosis via Sample-Level Capability Boundaries
arXiv · 29 sept. 2026
Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
arXiv · 6 oct. 2026
GPT-5.4 Repairs 48.8% of Qiskit Bugs Across Six Versions
Quantum Zeitgeist · 14 iul. 2026
[AINews] Gemini 4 Carbon tops leaderboards, launches for cyber defenders
Latent Space · 29 oct. 2026