Error Correction in Trained TTT Layers

Abstract:Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself establish how the memory revises an existing association. We study the structure and empirical role of prediction-derived updates in released TTT implementations. For dot-product objectives, we characterize the absence of a prediction-error residual and establish conditions under which inner-loop ascent and descent are equivalent after reparameterization. For the LayerNorm-based squared-error objective used by TTT-Linear and TTT-MLP, we derive the inner gradient and identify a prediction-derived component that cancels at the released initialization in the zero-epsilon limit. We then measure the full normalized prediction's contribution on three trained checkpoints. Removing this contribution while retaining the LayerNorm Jacobian gives a mean gradient norm ratio of 0.25 to 0.32 across the models. On 100,303 tokens from War and Peace, the same intervention with local per-token norm matching raises perplexity by 2.8-2.9% on TTT-Linear-760M and 1.3B, whereas freezing all fast-weight updates raises it by factors of 4.2-4.5. Separate synthetic tests illustrate the dependence of plain delta-rule overwriting on step size. The results distinguish a prediction-derived update from its behavioral value and show why implementation analysis and live interventions are both needed to assess test-time memory.

13 pages, 3 figures. Code and saved results included as ancillary material and at this https URL

Machine Learning (cs.LG); Computation and Language (cs.CL)

arXiv:2610.16268 [cs.LG]

(or arXiv:2610.16268v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.16268 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

recompute_and_check.py

requirements.txt

results/modal/ladder_linear-1.3b/ablation_results_linear13b.json

results/modal/ladder_linear-760m/ablation_results_linear760m.json

results/modal/probe_linear-1.3b.json

results/modal/probe_mlp-760m.json

results/overwrite_sweep_results.json

synthetic_notebook_record.json

validation_kit/LICENSE.ttt-lm-pytorch

validation_kit/bootstrap_spearman.py

validation_kit/convert_jax_to_pt.py

validation_kit/jax_ref_numpy.py

validation_kit/make_synthetic_jax_ckpt.py

validation_kit/run_ablation_ladder.py

validation_kit/run_delta_probe.py

validation_kit/run_validation.sh

validation_kit/test_conversion.py

validation_kit/ttt.py

validation_kit/ttt_arms.py

verified/conversion_test.log

verified/kernel_checks.json

verified/overwrite_verification.json

verified/source_identity.json

verified/verified_metrics.json

(22 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.