aiqbench analysis
The paper argues that a small, auditable agent harness tuned to a self-hostable open model can nearly match a large hosted agent baseline (83.8% vs. the Claude Code SDK's best, within 1.2 points) while being far smaller. For readers, the useful signal is the method: admitting only the tools, guardrails, and memory components that measurably improve accuracy on an expert-validated dataset. The results come from a single held-out dataset and a 3-page industry talk, so generalization to other investigative workloads remains untested.
Optimize the Harness, Not the Model: Fitting a Minimal Agent to a Self-Hostable LLM for Investigative Analysis
arXiv:2611.03434v1 Announce Type: new Abstract: Government intelligence teams need LLM agents that can analyze large, messy evidence collections, but security and cost constraints often rule out hosted frontier models and broad, hard-to-audit tool surfaces. We report CLEO: Constrained Loop for Environment Optimization, a method for fitting a minimal agent harness to a fixed, self-hostable LLM. Given an expert-validated dataset, CLEO admits only the tools, guardrails, memory policies, and loop components that improve accuracy while remaining general and auditable. On our held-out dataset, the CLEO harness reaches an 83.8% pass rate, within 1.2 points of the strongest out-of-the-box Claude Code SDK baseline, while being two orders of magnitude smaller.
Abstract:Government intelligence teams need LLM agents that can analyze large, messy evidence collections, but security and cost constraints often rule out hosted frontier models and broad, hard-to-audit tool surfaces. We report CLEO: Constrained Loop for Environment Optimization, a method for fitting a minimal agent harness to a fixed, self-hostable LLM. Given an expert-validated dataset, CLEO admits only the tools, guardrails, memory policies, and loop components that improve accuracy while remaining general and auditable. On our held-out dataset, the CLEO harness reaches an 83.8% pass rate, within 1.2 points of the strongest out-of-the-box Claude Code SDK baseline, while being two orders of magnitude smaller.
3 pages, 1 table. Industry Day talk at CIKM 2026, Rome, 7 November 2026
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multiagent Systems (cs.MA)
arXiv:2611.03434 [cs.AI]
(or arXiv:2611.03434v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2611.03434 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3799682.3839256 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 228 words