analiza aiqbench
OrionMoE-v1.0 introduces a sparse mixture-of-experts architecture specifically designed for tabular in-context learning, offering a more efficient alternative to dense foundation models. By selectively activating experts based on feature representations and dataset context, this model achieves competitive performance on small-context and high-dimensional benchmarks without requiring gradient-based adaptation. The approach highlights the potential of distribution-aware representations and synthetic causal priors for advancing specialized machine learning capabilities.
OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning
arXiv:2610.20418v1 Announce Type: new Abstract: Tabular foundation models enable in-context learning by predicting unseen observations from a labeled support table without task-specific parameter updates. Despite recent advances, most existing approaches rely on dense architectures that apply the same transformations across features and datasets. We introduce OrionMoE-Small, a compact tabular foundation model that combines distribution-aware representations with dataset-conditioned sparse mixture-of-experts computation. Built on a dual-axis Transformer architecture, OrionMoE-Small selectively activates experts along the feature axis, with routing decisions informed by both individual feature representations and the broader dataset context. This design enables adaptive allocation of computation across heterogeneous tabular problems while retaining a shared computational pathway across tasks. OrionMoE-Small is pretrained entirely on synthetic classification tasks generated from a structured causal prior and performs downstream prediction without gradient-based adaptation. Experiments across multiple tabular benchmarks demonstrate competitive performance against substantially larger foundation models, with particularly strong results on small-context and high-dimensional datasets. Further analysis reveals favorable relative performance under class imbalance and highlights the importance of feature composition.
Title:OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning
Abstract:Tabular foundation models enable in-context learning by predicting unseen observations from a labeled support table without task-specific parameter updates. Despite recent advances, most existing approaches rely on dense architectures that apply the same transformations across features and datasets. We introduce OrionMoE-Small, a compact tabular foundation model that combines distribution-aware representations with dataset-conditioned sparse mixture-of-experts computation. Built on a dual-axis Transformer architecture, OrionMoE-Small selectively activates experts along the feature axis, with routing decisions informed by both individual feature representations and the broader dataset context. This design enables adaptive allocation of computation across heterogeneous tabular problems while retaining a shared computational pathway across tasks. OrionMoE-Small is pretrained entirely on synthetic classification tasks generated from a structured causal prior and performs downstream prediction without gradient-based adaptation. Experiments across multiple tabular benchmarks demonstrate competitive performance against substantially larger foundation models, with particularly strong results on small-context and high-dimensional datasets. Further analysis reveals favorable relative performance under class imbalance and highlights the importance of feature composition.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.20418 [cs.LG]
(or arXiv:2610.20418v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.20418 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 267 cuvinte
Subiecte conexe
Xiaomi-TabLDM: A Tabular Foundation Model Technical Report
arXiv · 5 sept. 2026
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · 24 iul. 2026
Multi-Modal Spatio-Temporal Graph Neural Network with Mixture of Experts for Soil Organic Carbon Prediction
arXiv · 16 iun. 2026
Complete Guide to Thinking Machines Inkling
Analytics Vidhya · 20 iul. 2026