analiza aiqbench
This research evaluates "typed decision models," which output fixed probabilities rather than generated text, demonstrating they match LLM performance on tabular tasks like anomaly detection at a lower cost. However, the study highlights a critical robustness gap where model outputs shift significantly under minor rewrites of questions or table formatting, leaving reliability as an open problem for practical deployment.
Typed Decision Models on Tabular Data
arXiv:2610.19536v1 Announce Type: new Abstract: Typed decision models are language models that answer a question with a fixed set of possible answers by returning a probability for each answer instead of generating text. We study them on tabular data and compare four decision models with three large language models (LLMs) on row classification, fact verification and anomaly detection. Every request is sent in its original form and under 21 types of rewrite that keep its content, such as renaming the options or writing the table in another format. The decision models perform as well as the LLMs at a lower cost. Three of them are also far more robust to how the question is formulated. No model we test is robust to how the table is organized. For every model, at least a third of the row decisions change under some rewrite of the table or the question. The decisions to which a decision model gives low probability are the ones most likely to change. Decision models are a practical choice for tabular decisions, but their robustness remains an open problem.
Abstract:Typed decision models are language models that answer a question with a fixed set of possible answers by returning a probability for each answer instead of generating text. We study them on tabular data and compare four decision models with three large language models (LLMs) on row classification, fact verification and anomaly detection. Every request is sent in its original form and under 21 types of rewrite that keep its content, such as renaming the options or writing the table in another format. The decision models perform as well as the LLMs at a lower cost. Three of them are also far more robust to how the question is formulated. No model we test is robust to how the table is organized. For every model, at least a third of the row decisions change under some rewrite of the table or the question. The decisions to which a decision model gives low probability are the ones most likely to change. Decision models are a practical choice for tabular decisions, but their robustness remains an open problem.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2610.19536 [cs.CL]
(or arXiv:2610.19536v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.19536 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 277 cuvinte
Subiecte conexe
Fuzzy Distribution Modeling for Synthetic Tabular Data Generation with Causality Preservation
arXiv · 29 sept. 2026
Multimodal Attention-based Deep Learning for Emergency Triage with Electronic Health Records
arXiv · 21 iul. 2026
7 Machine Learning Algorithms That Still Matter
KDnuggets · 30 iul. 2026
Reviewing failure modes in science as we prepare for AI researchers
LessWrong · 8 oct. 2026