analiza aiqbench
This paper introduces RefineICL, a novel architecture for tabular foundation models that utilizes attention-gated updates to refine representations without modifying model parameters. By eliminating standard FFN layers in favor of low-rank feature interactions and typed memory, the method achieves significant performance gains on benchmarks like TabArena while reducing inference memory overhead compared to expanded feed-forward networks. The study provides theoretical derivation and empirical validation showing how these dynamic updates construct task-specific predictors directly from support data.
What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates
arXiv:2609.27679v1 Announce Type: new Abstract: What reusable computation should a tabular foundation model learn when every table defines a new supervised task? We develop in-situ representation refinement: support labels guide updates to the episode's representations, and these updates transfer to unlabeled queries without changing model parameters. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8. Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update, while preserving the query output, increases final query cross-entropy in all 72 tested episodes. Together, the derivation and interventions explain how attention-gated updates can construct a task-specific predictor in context.
Title:What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates
Abstract:What reusable computation should a tabular foundation model learn when every table defines a new supervised task? We develop in-situ representation refinement: support labels guide updates to the episode's representations, and these updates transfer to unlabeled queries without changing model parameters. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8. Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update, while preserving the query output, increases final query cross-entropy in all 72 tested episodes. Together, the derivation and interventions explain how attention-gated updates can construct a task-specific predictor in context.
Machine Learning (cs.LG)
arXiv:2609.27679 [cs.LG]
(or arXiv:2609.27679v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.27679 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 298 cuvinte
Subiecte conexe
OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning
arXiv · 15 oct. 2026
Introducing TabFM: A zero-shot foundation model for tabular data
Google Research · 30 iun. 2026
When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models
arXiv · 15 oct. 2026
Build Your Own Transaction Foundation Model for Financial Intelligence
NVIDIA · 16 iun. 2026