analiza aiqbench
This research evaluates open-weight large language models in repairing PDDL planning domains, demonstrating significant performance gains over symbolic baselines but highlighting critical reliability gaps where test constraint satisfaction remains inconsistent. The findings underscore that while LLMs can enhance automated model repair efficiency, they currently lack the robustness required for guaranteed correctness without additional verification mechanisms or human oversight.
LLM-Only PDDL Domain Repair with Open-Weight Models
arXiv:2608.17341v1 Announce Type: new Abstract: AI planning is concerned with finding a sequence of actions that achieves a specified goal. It relies on explicit models of the world, commonly represented in the Planning Domain Definition Language (PDDL). An active line of research investigates how errors in such models can be detected and repaired. For example, users may provide positive test plans that are solutions, and negative test plans that fail during execution. Automated repair methods then modify the PDDL model to satisfy these constraints. In this paper, we evaluate the ability of recent open-weight large language models to perform this repair task using an LLM-only approach. Our experiments show that the symbolic baseline achieves an $F_1$ score of $.49$, while the best-performing LLM reaches $.87$ with high reasoning effort, an absolute improvement of $.38$. However, that setting has a mean test pass rate of only $.82$, falling to $.06$ on the Thoughtful domain; even the best setting that includes the test traces reaches only $.92$. Thus, current open-weight models cannot guarantee satisfaction of the test constraints required for reliable automated model repair.
Abstract:AI planning is concerned with finding a sequence of actions that achieves a specified goal. It relies on explicit models of the world, commonly represented in the Planning Domain Definition Language (PDDL). An active line of research investigates how errors in such models can be detected and repaired. For example, users may provide positive test plans that are solutions, and negative test plans that fail during execution. Automated repair methods then modify the PDDL model to satisfy these constraints. In this paper, we evaluate the ability of recent open-weight large language models to perform this repair task using an LLM-only approach. Our experiments show that the symbolic baseline achieves an $F_1$ score of $.49$, while the best-performing LLM reaches $.87$ with high reasoning effort, an absolute improvement of $.38$. However, that setting has a mean test pass rate of only $.82$, falling to $.06$ on the Thoughtful domain; even the best setting that includes the test traces reaches only $.92$. Thus, current open-weight models cannot guarantee satisfaction of the test constraints required for reliable automated model repair.
Artificial Intelligence (cs.AI)
arXiv:2608.17341 [cs.AI]
(or arXiv:2608.17341v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.17341 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 271 cuvinte
Subiecte conexe
The Entangled Dimensions of Decision Theory
LessWrong · 30 iul. 2026
Models Can't Remember Their Training. Neither Can You.
LessWrong · 19 iul. 2026
Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding
arXiv · 21 iul. 2026
Identity Continuity in Long-Term Embodied AI Relationships: From Agent-Specific Identity Representation to Identity-Continuity Appraisal
arXiv · 22 sept. 2026