aiqbench analysis
This research proposes a symplectic phase-space lifting technique to bypass the single-layer induction obstruction identified by Sanford, Hsu, and Telgarsky in transformer architectures. By applying a specific shear transformation to post-RoPE query and key streams, the method enables one-step causal bilinear designs to achieve higher-order induction without adding parameters or modifying KV cache structures. The approach offers a theoretically grounded pathway to improve long-context modeling capabilities while maintaining computational efficiency through fused kernels and minimal overhead.
Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction
arXiv:2609.32319v1 Announce Type: new Abstract: We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a $z$-transform of joint order $(0,0)$, whereas the induction discriminator requires key-side order $\geq 1$. Applying the symplectic upper shear $M_\gamma:(q,p)\mapsto(q+\gamma p,p)$ to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit $T_4$-only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at $r=0.9876$ with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the $T_3$ channel moves $\gamma_c$ at $d_k=64$ from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is $6d$ FLOPs per token per layer, INT8 headroom grows by at most $\log_2(1+2\gamma)$ bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at $\gamma=0.80$ in a mean of 717 steps, against 2/3 seeds and 2700 steps at $\gamma=0$. Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.
Abstract:We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a $z$-transform of joint order $(0,0)$, whereas the induction discriminator requires key-side order $\geq 1$. Applying the symplectic upper shear $M_\gamma:(q,p)\mapsto(q+\gamma p,p)$ to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit $T_4$-only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at $r=0.9876$ with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the $T_3$ channel moves $\gamma_c$ at $d_k=64$ from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is $6d$ FLOPs per token per layer, INT8 headroom grows by at most $\log_2(1+2\gamma)$ bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at $\gamma=0.80$ in a mean of 717 steps, against 2/3 seeds and 2700 steps at $\gamma=0$. Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.
Machine Learning (cs.LG); Materials Science (cond-mat.mtrl-sci); Artificial Intelligence (cs.AI)
arXiv:2609.32319 [cs.LG]
(or arXiv:2609.32319v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.32319 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
PSA_notebooks_41/README.txt
PSA_notebooks_41/notebooks/Appendix_C.ipynb
PSA_notebooks_41/notebooks/Appendix_D.ipynb
PSA_notebooks_41/notebooks/Appendix_E.ipynb
PSA_notebooks_41/notebooks/Appendix_F1.ipynb
PSA_notebooks_41/notebooks/Appendix_F2.ipynb
PSA_notebooks_41/notebooks/Appendix_F3.ipynb
PSA_notebooks_41/notebooks/Appendix_G.ipynb
PSA_notebooks_41/notebooks/Appendix_H.ipynb
PSA_notebooks_41/notebooks/Appendix_I_Mechanistic_Visualization.ipynb
PSA_notebooks_41/notebooks/Appendix_J_CoT_Reasoning_NB1.ipynb
PSA_notebooks_41/notebooks/Appendix_J_CoT_Reasoning_NB2.ipynb
PSA_notebooks_41/notebooks/Appendix_K_Real_World_Reasoning.ipynb
PSA_notebooks_41/notebooks/Appendix_L_Multi_Task_Validation.ipynb
PSA_notebooks_41/notebooks/Appendix_M_Multi_Difficulty_Validation.ipynb
PSA_notebooks_41/notebooks/Appendix_N_NB_1.ipynb
PSA_notebooks_41/notebooks/Appendix_N_NB_2.ipynb
PSA_notebooks_41/notebooks/Appendix_N_NB_3.ipynb
PSA_notebooks_41/notebooks/Appendix_N_NB_4.ipynb
PSA_notebooks_41/notebooks/Appendix_N_NB_5.ipynb
PSA_notebooks_41/notebooks/Appendix_O_P.ipynb
PSA_notebooks_41/notebooks/Appendix_P.ipynb
PSA_notebooks_41/notebooks/Appendix_Q_Final_v2.ipynb
PSA_notebooks_41/notebooks/Appendix_Q_Stability.ipynb
PSA_notebooks_41/notebooks/Appendix_Q_bracket_dk128.ipynb
PSA_notebooks_41/notebooks/Appendix_Q_cutoff_decisive.ipynb
PSA_notebooks_41/notebooks/Appendix_R2_Mechanism.ipynb
PSA_notebooks_41/notebooks/Appendix_R_NB_1.ipynb
PSA_notebooks_41/notebooks/Appendix_R_NB_2.ipynb
PSA_notebooks_41/notebooks/Experiment_16_Single_Layer_Induction.ipynb
PSA_notebooks_41/notebooks/Experiment_17_Scaling_Law_Final.ipynb
PSA_notebooks_41/notebooks/Experiment_18_Granular_Scaling_Law.ipynb
PSA_notebooks_41/notebooks/PSA_Certificates.ipynb
PSA_notebooks_41/notebooks/PSA_Induction_Validation_Suite_v7.ipynb
PSA_notebooks_41/notebooks/PSA_Induction_Validation_Suite_v8.ipynb
PSA_notebooks_41/notebooks/PSA_Induction_Validation_Suite_v9.ipynb
PSA_notebooks_41/notebooks/PSA_MC_dk_sweep_with_results.ipynb
PSA_notebooks_41/notebooks/PSA_T_axis_condensed_validation.ipynb
PSA_notebooks_41/notebooks/PSA_control_battery_results.ipynb
PSA_notebooks_41/notebooks/PSA_spectral_diagnostic_rerun.ipynb
PSA_notebooks_41/notebooks/PSA_stability_characterisation_results.ipynb
PSA_notebooks_41/notebooks/PSA_trained_dk_overlay_with_results.ipynb
PSA_notebooks_41/results/Appendix_Q_Final_v2/q_final_ledger.jsonl
PSA_notebooks_41/results/Appendix_Q_Final_v2/q_final_results.json
PSA_notebooks_41/results/Appendix_Q_bracket_dk128/q_bracket_ledger.jsonl
PSA_notebooks_41/results/Appendix_Q_bracket_dk128/q_bracket_results.json
PSA_notebooks_41/results/Appendix_Q_cutoff_decisive/q_cutoff_ledger.jsonl
PSA_notebooks_41/results/Appendix_Q_cutoff_decisive/q_cutoff_results.json
PSA_notebooks_41/results/supplementary_diagnostics/diag_results_collapse.json
PSA_notebooks_41/results/supplementary_diagnostics/diag_results_v6.json
PSA_notebooks_41/results/supplementary_diagnostics/q2_diagnostic.pdf
PSA_notebooks_41/results/supplementary_diagnostics/q3_window.pdf
generators_and_audit/audit_v88.py
generators_and_audit/make_fig12_band.py
generators_and_audit/make_fig13_cutoff.py
generators_and_audit/make_fig1_storyboard.py
generators_and_audit/make_fig3_bode.py
generators_and_audit/make_rem_fig.py
q_bracket_results.json
q_cutoff_results.json
q_final_results.json
(56 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 450 words
Related coverage
Bluesky’s new CEO wants a big tent, not a bubble
The Verge · Aug 3, 2026
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · Oct 8, 2026
Enterprise Quantum Cybersecurity, a Practical Guide for 2026
Quantum Zeitgeist · Jul 14, 2026
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · Oct 8, 2026