analiza aiqbench
This meta-evaluation of table LLMs reveals that base model selection exerts a significantly stronger influence on performance than the specific training datasets used for instruction tuning. Despite these findings, the study underscores that generalization and reasoning capabilities in table modeling remain unresolved challenges requiring targeted future research. For practitioners, this suggests prioritizing foundation model architecture over dataset curation when optimizing for tabular tasks.
What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects
arXiv:2501.14717v2 Announce Type: replace Abstract: Table modeling has progressed for decades. In this work, we revisit this trajectory and highlight emerging challenges in the LLM era, particularly the paradox of choice: the difficulty of attributing performance gains amid diverse base models and training sets in the context of table instruction tuning. We replicate four table LLMs by instruction-tuning three foundation models on four existing datasets, yielding 12 models. We then evaluate these models across 16 table benchmarks. Our study is the first to quantitatively disentangle the effects of training data and base model selection, revealing that base model choice plays a more dominant role than the training data itself. Generalization and reasoning remain challenging, inviting future effort on table modeling. Based on our findings, we share our thoughts on the future directions for table modeling.
Title:What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects
Abstract:Table modeling has progressed for decades. In this work, we revisit this trajectory and highlight emerging challenges in the LLM era, particularly the paradox of choice: the difficulty of attributing performance gains amid diverse base models and training sets in the context of table instruction tuning. We replicate four table LLMs by instruction-tuning three foundation models on four existing datasets, yielding 12 models. We then evaluate these models across 16 table benchmarks. Our study is the first to quantitatively disentangle the effects of training data and base model selection, revealing that base model choice plays a more dominant role than the training data itself. Generalization and reasoning remain challenging, inviting future effort on table modeling. Based on our findings, we share our thoughts on the future directions for table modeling.
Computation and Language (cs.CL)
arXiv:2501.14717 [cs.CL]
(or arXiv:2501.14717v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2501.14717 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 236 cuvinte