analiza aiqbench
This arXiv paper is about training techniques for language-model scoring heads, specifically whether pooling choices and selective low-rank adaptation that work for direct answer selection also transfer to entailment-based reranking. The reported gains are modest and measured on a benchmark averaged over eleven configurations from seven sources, so they should be read as incremental, pending independent replication. For practitioners, the practical takeaway is that fine-tuning fewer parameters (attention and DeltaNet projections, no feed-forward adapters) can match or beat an all-module baseline, which lowers training cost.
Improving JEV Entailment Reranking with Span Readouts and Selective Adaptation
arXiv:2610.35654v1 Announce Type: new Abstract: JEV decision models pair a language model backbone with a lightweight scoring head to select among natural-language options. Direct selection compares candidate answers within one request, whereas entailment reranking evaluates each candidate in a separate natural language inference (NLI) request over fixed relation options. We examine whether the readout and adaptation choices used for direct selection remain effective for cross-request reranking. We combine a mean-span readout with selective backbone adaptation. Mean-span pooling replaces the option-ending pointer with an average of the contextualized states inside each NLI option, without adding head parameters. Selective adaptation retains low-rank updates in attention and gated DeltaNet projections while removing feed-forward adapters. On MC11, which averages accuracy over eleven reranking configurations from seven sources, selective adaptation improves NeoHorse-1-4B accuracy over the all-module baseline while cutting trainable parameters by more than half, and mean-span pooling yields further consistent gains at the same parameter count.
Abstract:JEV decision models pair a language model backbone with a lightweight scoring head to select among natural-language options. Direct selection compares candidate answers within one request, whereas entailment reranking evaluates each candidate in a separate natural language inference (NLI) request over fixed relation options. We examine whether the readout and adaptation choices used for direct selection remain effective for cross-request reranking. We combine a mean-span readout with selective backbone adaptation. Mean-span pooling replaces the option-ending pointer with an average of the contextualized states inside each NLI option, without adding head parameters. Selective adaptation retains low-rank updates in attention and gated DeltaNet projections while removing feed-forward adapters. On MC11, which averages accuracy over eleven reranking configurations from seven sources, selective adaptation improves NeoHorse-1-4B accuracy over the all-module baseline while cutting trainable parameters by more than half, and mean-span pooling yields further consistent gains at the same parameter count.
25 pages, 3 figures. Project page: this https URL
Computation and Language (cs.CL)
arXiv:2610.35654 [cs.CL]
(or arXiv:2610.35654v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.35654 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 251 cuvinte
Subiecte conexe
Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics
arXiv · 19 aug. 2026
Solipsistic Superintelligence is Unlikely to be Cooperative
arXiv · 3 iun. 2026
Nereus: Adaptive Parallelism for LLM Post-Training
arXiv · 29 sept. 2026
Knowing When Not to Reuse: Conditional Experience Transfer in Autonomous LLM Post-Training
arXiv · 28 aug. 2026