aiqbench analysis
The paper argues that language models struggle with high-level instructions like "stack blocks in a pyramid," and that an explicit domain-specific language layer can close much of that gap. Code generation over a human-curated DSL reaches human-level performance on the Hexagons benchmark, with the largest gains at higher abstraction levels, and it also helps on Text2CAD. Model-induced DSLs recover only part of that benefit and are inconsistent across runs, so automatically learning the representation remains the open problem.
From Abstraction to Action: DSLs for Natural Language Abstraction Interpretation and Execution
arXiv:2610.28311v1 Announce Type: new Abstract: Natural-language instructions often rely on high-level abstractions, such as asking an agent to ``stack blocks in a pyramid.'' Interpreting abstract instructions requires models to both identify the abstraction intended by the speaker, and unroll it into the many concrete actions that instantiate it. Language models struggle here: prior work has shown that performance degrades as the level of abstraction in the instruction rises. To bridge the gap between natural language abstraction and concrete actions, we propose to borrow from coding environments the notion of a {\em domain-specific language} (DSL), an explicit representation layer whose documented primitives name recurring abstractions that expand into concrete actions. We demonstrate the efficacy of DSLs for NL interpretation on \textsc{Hexagons}, a visually-grounded instruction-following task graded for abstraction. We further ask whether language models can induce such DSLs themselves from domain data. Our results show that while code generation consistently improves over direct inference, code generation over a human-curated DSL yields dramatic gains, up to 10 absolute F1 points, reaching the human skyline, with gains concentrated at {\em higher} abstraction levels. We confirm this pattern on the \textsc{Text2CAD} benchmark. At the same time, model-induced DSLs recover only part of this gain, remaining inconsistent across runs and short of their human-curated counterpart. Overall, the right representation closes much of the gap caused by high abstraction --- but inducing that layer of representation automatically is where models still fall short, and where we see a strong opportunity for the community to make progress.
Abstract:Natural-language instructions often rely on high-level abstractions, such as asking an agent to ``stack blocks in a pyramid.'' Interpreting abstract instructions requires models to both identify the abstraction intended by the speaker, and unroll it into the many concrete actions that instantiate it. Language models struggle here: prior work has shown that performance degrades as the level of abstraction in the instruction rises. To bridge the gap between natural language abstraction and concrete actions, we propose to borrow from coding environments the notion of a {\em domain-specific language} (DSL), an explicit representation layer whose documented primitives name recurring abstractions that expand into concrete actions. We demonstrate the efficacy of DSLs for NL interpretation on \textsc{Hexagons}, a visually-grounded instruction-following task graded for abstraction. We further ask whether language models can induce such DSLs themselves from domain data. Our results show that while code generation consistently improves over direct inference, code generation over a human-curated DSL yields dramatic gains, up to 10 absolute F1 points, reaching the human skyline, with gains concentrated at {\em higher} abstraction levels. We confirm this pattern on the \textsc{Text2CAD} benchmark. At the same time, model-induced DSLs recover only part of this gain, remaining inconsistent across runs and short of their human-curated counterpart. Overall, the right representation closes much of the gap caused by high abstraction --- but inducing that layer of representation automatically is where models still fall short, and where we see a strong opportunity for the community to make progress.
Computation and Language (cs.CL)
arXiv:2610.28311 [cs.CL]
(or arXiv:2610.28311v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.28311 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 340 words
Related coverage
SAIL: Sound Abstract Interpreters with LLMs
arXiv · Jun 3, 2026
Automatic Tensor-level Execution of Abstract Interpreters for Neural Networks
arXiv · Oct 19, 2026
Temporally Interpretable Differentiable Decision Trees
arXiv · Oct 8, 2026
Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Units
arXiv · Jun 9, 2026