aiqbench analysis
This research addresses the high inference costs and lack of interpretability in applying Large Language Models directly to tabular data by distilling LLM knowledge into efficient decision trees. The proposed three-stage framework generates rules rather than full trees, achieving superior accuracy and transparency with reduced prompting overhead in few-shot learning scenarios. This work offers a practical pathway to combine the world knowledge of LLMs with the speed and clarity of traditional tree models for real-world tabular datasets.
From Prompts to Trees: Effective LLM-Guided Tree Generation for Few-Shot Tabular Classification
arXiv:2610.10227v1 Announce Type: cross Abstract: While Large Language Models (LLMs) possess rich world knowledge and impressive generalization capabilities, their direct application to tabular data classification is hindered by high inference costs and limited interpretability. In contrast, decision trees are fast and transparent but often underperform in low-data regimes. In this work, we propose a novel framework that bridges these paradigms by distilling LLM knowledge into interpretable decision trees under a few-shot learning setting. Instead of directly prompting the LLM to generate full trees, which is often unstable and inefficient, we develop a three-stage paradigm that prompts the LLM to generate rules and organize the rules into a tree. Experiments on multiple real-world tabular datasets demonstrate that our method achieves superior accuracy and interpretability with significantly lower prompting overhead compared to existing baselines.
Abstract:While Large Language Models (LLMs) possess rich world knowledge and impressive generalization capabilities, their direct application to tabular data classification is hindered by high inference costs and limited interpretability. In contrast, decision trees are fast and transparent but often underperform in low-data regimes. In this work, we propose a novel framework that bridges these paradigms by distilling LLM knowledge into interpretable decision trees under a few-shot learning setting. Instead of directly prompting the LLM to generate full trees, which is often unstable and inefficient, we develop a three-stage paradigm that prompts the LLM to generate rules and organize the rules into a tree. Experiments on multiple real-world tabular datasets demonstrate that our method achieves superior accuracy and interpretability with significantly lower prompting overhead compared to existing baselines.
Accepted to EMNLP 2026 Main as an oral presentation. Code available: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.10227 [cs.LG]
(or arXiv:2610.10227v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.10227 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 242 words
Related coverage
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
AWS · Aug 12, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
Serve Kimi-K2.5-MXFP4 on MI355X with ATOM
AMD · Jul 23, 2026