analiza aiqbench
The paper argues that selective state-space models like Mamba can misalign their memory timescales when input data shift, because the step size and state matrix only enter the model through their product. Its proposed fix, PALM, ties both parameters to physical units. Reported gains on fMRI and weather forecasting are measured against parameter-matched baselines, and the authors attribute them to placement of memory in physical time rather than model size. These are preprint results on two domains, so readers should check whether the anchoring generalizes beyond them.
PALM: Physically-Anchored Latent-timescale Mamba for Stable Representations under Distribution Shift
arXiv:2605.16351v2 Announce Type: replace Abstract: Scientific foundation models are built to reuse representations across datasets and acquisition protocols, yet their representations degrade under distribution shift. We trace one source of this failure to selective state-space models (SSMs). Their memory depends on the step Delta and the state matrix A only through their product, so data do not identify the physical timescale of memory, and under a shift the resulting memory no longer matches the signal's timescales. We call this failure memory-timescale misalignment. To resolve it, the split between Delta and A must be tied to physical units. We therefore propose Physically-Anchored Latent-timescale Mamba (PALM), which disentangles A from Delta by anchoring both to acquisition units: Delta through spectrum-estimated knee frequencies and A through a physical memory range. We evaluate PALM against parameter-matched baselines in two domains, under temporal-context truncation, low-resource transfer, and brain-state shift on fMRI and under held-out-station forecasting on Weather-5K. Under fMRI context truncation with run-level spectral calibration, PALM keeps its latent timescales in physical units and retains 11% higher representation similarity than Mamba2. At geographically isolated weather stations never seen during training, PALM achieves the lowest error on every variable and horizon, 15.2% below the best baseline on average, and outperforms zero-shot forecasting foundation models. Across both domains, the gains come from where the backbone places memory in physical time rather than from model size. We therefore argue that physically anchored latent timescales are a design requirement for scientific foundation models that must transfer across acquisition conditions.
Abstract:Scientific foundation models are built to reuse representations across datasets and acquisition protocols, yet their representations degrade under distribution shift. We trace one source of this failure to selective state-space models (SSMs). Their memory depends on the step Delta and the state matrix A only through their product, so data do not identify the physical timescale of memory, and under a shift the resulting memory no longer matches the signal's timescales. We call this failure memory-timescale misalignment. To resolve it, the split between Delta and A must be tied to physical units. We therefore propose Physically-Anchored Latent-timescale Mamba (PALM), which disentangles A from Delta by anchoring both to acquisition units: Delta through spectrum-estimated knee frequencies and A through a physical memory range. We evaluate PALM against parameter-matched baselines in two domains, under temporal-context truncation, low-resource transfer, and brain-state shift on fMRI and under held-out-station forecasting on Weather-5K. Under fMRI context truncation with run-level spectral calibration, PALM keeps its latent timescales in physical units and retains 11% higher representation similarity than Mamba2. At geographically isolated weather stations never seen during training, PALM achieves the lowest error on every variable and horizon, 15.2% below the best baseline on average, and outperforms zero-shot forecasting foundation models. Across both domains, the gains come from where the backbone places memory in physical time rather than from model size. We therefore argue that physically anchored latent timescales are a design requirement for scientific foundation models that must transfer across acquisition conditions.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.16351 [cs.LG]
(or arXiv:2605.16351v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.16351 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 340 cuvinte