analiza aiqbench
This arXiv paper introduces SERIAL-7, a Korean-English benchmark that tests whether LLMs produce structured outputs that programs can reliably consume, checking format, structure, and task semantics with executable tests. Results across ten models vary widely (REALIZE-L3 HSR from 42.0% to 93.9%), and stricter "Full Compliance" checks reveal artifact-level failures even when final answers are correct. For developers building LLM-driven pipelines, the takeaway is that semantic accuracy alone is not enough, and executable validation of outputs is needed.
SERIAL-7: Deriving, Realizing, and Preserving Machine-Verifiable Outputs Across Representations
arXiv:2610.41308v1 Announce Type: new Abstract: Large language models increasingly generate structured outputs consumed by programs and downstream systems, where semantic correctness alone does not guarantee reliable execution. We introduce SERIAL-7, a Korean-English benchmark built around an executable verification framework for machine-verifiable generation across seven representations. Using 177 validated canonical artifacts, SERIAL-7 separates semantic derivation (Derive), constrained artifact generation (Realize), and cross-representation preservation (Preserve). Its 43 atomic representation constraints are paired with executable checks over format, structure, and task-specific semantics, enabling progressively stricter verification from representation compliance to full artifact validity. Across ten models, REALIZE-L3 HSR ranges from 42.0% to 93.9%, while Full Compliance exposes additional artifact-level failures beyond correct final answers and valid representations. These results show that reliable structured generation requires explicit, executable verification of how artifacts are realized and preserved across representations.
Abstract:Large language models increasingly generate structured outputs consumed by programs and downstream systems, where semantic correctness alone does not guarantee reliable execution. We introduce SERIAL-7, a Korean-English benchmark built around an executable verification framework for machine-verifiable generation across seven representations. Using 177 validated canonical artifacts, SERIAL-7 separates semantic derivation (Derive), constrained artifact generation (Realize), and cross-representation preservation (Preserve). Its 43 atomic representation constraints are paired with executable checks over format, structure, and task-specific semantics, enabling progressively stricter verification from representation compliance to full artifact validity. Across ten models, REALIZE-L3 HSR ranges from 42.0% to 93.9%, while Full Compliance exposes additional artifact-level failures beyond correct final answers and valid representations. These results show that reliable structured generation requires explicit, executable verification of how artifacts are realized and preserved across representations.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.41308 [cs.AI]
(or arXiv:2610.41308v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.41308 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 227 cuvinte