Can Decision Models (Like Jev) Plan Ahead?

Introducion Can decision models like Jev plan ahead? If so, how can we quantify their planning ability? This is the question that BeelineBench seeks to answer. In this article, I'll explain my methodology and share a few interesting results from the benchmark. Methodology The core idea is simple. We present a decision model with a problem domain (like Blocks world) and a list of available actions, and ask it to select the next action in order to solve the problem. We repeat until the decision model has reached a solution, or has reached an evaluation cap. If the decision model explores few unnecessary moves before reaching a solution, the intuition is that the decision model is exhibiting an ability to plan ahead while selecting an action. To compute the beeline metric, we first count the number of nodes along a shortest path found using a breadth-first search. We then divide by the number of nodes explored by the decision model while finding a solution. score = (shortest path + 1) / nodes explored with the model A perfect score is 1, meaning that the decision model reached a solution by exploring exactly the minimum number of nodes. We include baselines against random-choice and a classic heuristic as deciders. N=100 trials are drawn for each problem domain, and 95% confidence intervals are bootstrapped from the scores. We also compute a number of ancillary metrics and diagnostics. Results - Beeline Scores In four of seven problem domains, at least one decider model was able to beat the heuristic-based baseline. All included decider models beat random choice on all problems. Rush Hour was by far the most difficult problem for decider models. The decider models seemed to handle Wikispeedia and "Keys and Doors" quite well. Cost and Efficiency A few models appear frequently on the frontier, including Mercury Decide and pplx-decider 1.0. Limitations We acknowledge that decider models are highly sensitive to how the problem domain is represented textually. In the included problem domains, we attempted to choose a "natural" or "obvious" representation. We felt this was a better choice than attempting to jointly optimize performance across models by experimenting with textual representations, as this sort of ad hoc experimentation ran the risk of invalidating the benchmark. For now, we consider the ability of the model to understand a reasonable textual representation to be 'part of the benchmark'. GPT-6 Luna had a high refusal rate for some problem domains, contaminating its score. GPT-6 Luna appears to refuse to answer when the next choice is unclear, but this is an informal observation rather than a validated finding. Jev has a cap of 255 options per decision. In order to respect Jev's cap in an unbiased way across all deciders (including baselines), we randomly drop options until we are under the cap. In future evaluations, we may limit this behavior to Jev and consider it a handicap specific to Jev. Next Steps We plan on including additional models and curating our problem domains. Expense is a factor, so we appreciate donations through Github. The Repo BeelineBench is a portable, configurable, reproducible, and re-randomizable benchmark. Each problem domain consists of 100 trials. The trials are randomly drawn from each domain, with an optional master seed for re-randomization. 95% confidence intervals are bootstrapped from each model's result for each problem domain. The bench is designed to plug in new models, new problems, and to gracefully handle restarts. A PyPi package and Docker containers are included. Discuss

Full text under the publisher's open licence · 552 words