analiza aiqbench
This paper proposes RT-J, an 85M-parameter foundation model for relational databases that makes accurate predictions from only hundreds of in-context examples, rather than the tens of thousands that existing models need. Its claimed 10x sample-efficiency gain over feature-engineering pipelines is measured on RelBench regression tasks, so readers should check how it generalizes beyond that benchmark. The work's main contributions are a large corpus of web-crawled databases, a random-walk retriever for selecting context, and test-time scaling, which are useful reference points for anyone working on tabular or structured-data ML.
RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions
arXiv:2610.17867v1 Announce Type: new Abstract: Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.
Abstract:Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.
Accepted to NeurIPS 2026. Website: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Databases (cs.DB)
arXiv:2610.17867 [cs.LG]
(or arXiv:2610.17867v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.17867 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 268 cuvinte
Subiecte conexe
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · 29 sept. 2026
[AINews] The Future of Latent Space
Latent Space · 25 sept. 2026