analiza aiqbench
This research introduces "compiled AI," a systems approach where large language models generate deterministic code artifacts during an initial compilation phase, eliminating runtime model invocations for subsequent workflow execution. By constraining generation to validated templates within high-stakes domains like healthcare, the paradigm trades probabilistic flexibility for critical gains in auditability, security, and cost efficiency while significantly reducing token consumption. The study validates this architecture through rigorous evaluation on function-calling tasks and document intelligence benchmarks, demonstrating its viability for production enterprise environments where reliability is paramount.
Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation
arXiv:2604.05150v2 Announce Type: replace-cross Abstract: We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation. This paradigm has antecedents in prior work on declarative pipeline optimization (DSPy) and hybrid neural-symbolic planning (LLM+P); our contribution is a systems-oriented study of its application to high-stakes enterprise workflows, with particular emphasis on healthcare settings where reliability and auditability are critical. By constraining generation to narrow business-logic functions embedded in validated templates, compiled AI trades runtime flexibility for predictability, auditability, cost efficiency, and reduced security exposure. We introduce (i) a system architecture for constrained LLM-based code generation, (ii) a four-stage generation-and-validation pipeline that converts probabilistic model output into production-ready code artifacts, and (iii) an evaluation framework measuring operational metrics including token amortization, determinism, reliability, security, and cost. We evaluate on two task types: function-calling (BFCL, n=400) and document intelligence (DocILE, n=5,680 invoices). On function-calling, compiled AI achieves 96% task completion with zero execution tokens, breaking even with runtime inference at approximately 17 transactions and reducing token consumption by 57x at 1,000 transactions. On document intelligence, our Code Factory variant matches Direct LLM on key field extraction (KILE: 80.0%) while achieving the highest line item recognition accuracy (LIR: 80.4%). Security evaluation across 135 test cases demonstrates 96.7% accuracy on prompt injection detection and 87.5% on static code safety analysis with zero false positives.
Authors:Geert Trooskens (1), Aaron Karlsberg (1), Anmol Sharma (1), Lamara De Brouwer (1), Max Van Puyvelde (2), Matthew Young (1), John Thickstun (3), Gil Alterovitz (4), Walter A. De Brouwer (2) ((1) XY.AI Labs, Palo Alto, CA, (2) Stanford University School of Medicine, Stanford, CA, (3) Cornell University, Ithaca, NY, (4) Brigham and Women's Hospital / Harvard Medical School, Boston, MA)
Abstract:We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation. This paradigm has antecedents in prior work on declarative pipeline optimization (DSPy) and hybrid neural-symbolic planning (LLM+P); our contribution is a systems-oriented study of its application to high-stakes enterprise workflows, with particular emphasis on healthcare settings where reliability and auditability are critical. By constraining generation to narrow business-logic functions embedded in validated templates, compiled AI trades runtime flexibility for predictability, auditability, cost efficiency, and reduced security exposure. We introduce (i) a system architecture for constrained LLM-based code generation, (ii) a four-stage generation-and-validation pipeline that converts probabilistic model output into production-ready code artifacts, and (iii) an evaluation framework measuring operational metrics including token amortization, determinism, reliability, security, and cost. We evaluate on two task types: function-calling (BFCL, n=400) and document intelligence (DocILE, n=5,680 invoices). On function-calling, compiled AI achieves 96% task completion with zero execution tokens, breaking even with runtime inference at approximately 17 transactions and reducing token consumption by 57x at 1,000 transactions. On document intelligence, our Code Factory variant matches Direct LLM on key field extraction (KILE: 80.0%) while achieving the highest line item recognition accuracy (LIR: 80.4%). Security evaluation across 135 test cases demonstrates 96.7% accuracy on prompt injection detection and 87.5% on static code safety analysis with zero false positives.
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
I.2.2; I.2.7; D.2.11
arXiv:2604.05150 [cs.SE]
(or arXiv:2604.05150v2 [cs.SE] for this version)
From: Geert Trooskens Ph.D. [view email] [v1] Mon, 6 Apr 2026 20:25:20 UTC (18 KB) [v2] Fri, 31 Jul 2026 14:32:47 UTC (23 KB)
Text integral sub licența deschisă a publicației · 333 cuvinte
