analiza aiqbench
This research addresses the critical limitation of fixed domain sizes in transformer-based neural operators, proposing a method for zero-shot generalization to significantly larger spatial domains without architectural changes. By integrating decomposable bias into attention logits with rotary positional embeddings, the approach achieves improved performance on partial differential equation benchmarks and industrial atmospheric flow applications. The work advances machine learning capabilities for solving complex physics problems at scale by ensuring robust inference across varying domain extents.
Zero-shot generalization of transformer neural operators to larger domains
arXiv:2606.14597v1 Announce Type: new Abstract: Transformer-based neural operators have shown remarkable performance for approximating solution operators of partial differential equations on complex geometries. However, existing approaches implicitly assume a fixed domain size, which limits their ability to generalize at inference. In this work, we investigate domain extension, namely zero-shot inference on spatial domains that are significantly larger than those encountered during training. We argue that this setting fundamentally requires spatial locality and translation equivariance. We propose to implement this locality via a decomposable bias in the attention logits computation, enabling finely controllable locality while remaining fully decomposable into query-key inner products and directly compatible with optimized attention kernels. Combined with rotary positional embeddings, it enables expressive embeddings with controllable spatial support without altering the transformer architecture. We empirically show that our approach substantially improves zero-shot generalization to larger domains across two PDE benchmarks and a 3D industrial atmospheric flow application. Our code and datasets are available at https://github.com/cerea-daml/domain-extension.
Abstract:Transformer-based neural operators have shown remarkable performance for approximating solution operators of partial differential equations on complex geometries. However, existing approaches implicitly assume a fixed domain size, which limits their ability to generalize at inference. In this work, we investigate domain extension, namely zero-shot inference on spatial domains that are significantly larger than those encountered during training. We argue that this setting fundamentally requires spatial locality and translation equivariance. We propose to implement this locality via a decomposable bias in the attention logits computation, enabling finely controllable locality while remaining fully decomposable into query-key inner products and directly compatible with optimized attention kernels. Combined with rotary positional embeddings, it enables expressive embeddings with controllable spatial support without altering the transformer architecture. We empirically show that our approach substantially improves zero-shot generalization to larger domains across two PDE benchmarks and a 3D industrial atmospheric flow application. Our code and datasets are available at this https URL.
Machine Learning (cs.LG)
arXiv:2606.14597 [cs.LG]
(or arXiv:2606.14597v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.14597 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 249 cuvinte
Subiecte conexe
Zero-shot generalization of transformer neural operators to larger domains
arXiv · 2 oct. 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers
Hugging Face · 26 aug. 2026
The Entangled Dimensions of Decision Theory
LessWrong · 30 iul. 2026