aiqbench analysis
The paper shows that high accuracy on asking whether two programs are semantically equivalent overstates how well language models understand code changes. Models that score above 94% on equivalence diagnose only about 54% of the underlying semantic changes, and reconstructing multi-step change sequences is much harder still, with the best model reaching 39% and none exceeding 1.11% at five changes. For readers evaluating code-reasoning models, the takeaway is that benchmark design matters: yes/no tests can hide failure modes that matter for real code review and refactoring.
SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code
arXiv:2610.41562v1 Announce Type: cross Abstract: Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.
Abstract:Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.
Software Engineering (cs.SE); Computation and Language (cs.CL)
arXiv:2610.41562 [cs.SE]
(or arXiv:2610.41562v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2610.41562 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 296 words
Related coverage
Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
arXiv · Aug 24, 2026
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
arXiv · Jun 9, 2026
Improving LLM Interpretability with User-Centric Chain-of-Thought Reasoning
arXiv · Aug 28, 2026
Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
arXiv · Aug 5, 2026