aiqbench analysis
TAFFY introduces a novel tabular foundation model architecture that leverages synthetic task diversity and iterative transformer refinement to enhance in-context learning capabilities. By dynamically modulating context integration based on task conditions, the approach achieves superior performance across multiple classification and regression benchmarks compared to existing methods. This work advances the state of pretraining strategies for structured data, offering a significant step forward for models designed to handle complex tabular relationships without extensive fine-tuning.
TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity
arXiv:2610.07559v1 Announce Type: new Abstract: Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.
Abstract:Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.
Machine Learning (cs.LG)
arXiv:2610.07559 [cs.LG]
(or arXiv:2610.07559v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.07559 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 268 words
Related coverage
OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning
arXiv · Oct 15, 2026
TabGift: A Context-Efficient Foundation Model for Tabular Data Generation
arXiv · Oct 15, 2026
Build Your Own Transaction Foundation Model for Financial Intelligence
NVIDIA · Jun 16, 2026
In-Context Learning for Latent Space Bayesian Optimization
arXiv · Jun 9, 2026