analiza aiqbench
This research introduces LEDOM, a reverse autoregressive language model that conditions on future context to predict the past, yielding distinct capabilities like abductive inference and improved reasoning verification. By combining forward likelihood with reverse posterior estimates via noisy channel duality, the proposed Reverse Reward method effectively penalizes hallucinated chains while boosting performance on math benchmarks such as AIME and AMC. The release of open-source models and code marks a significant step toward bidirectional scoring architectures that enhance model reliability without relying solely on left-to-right training paradigms.
LEDOM: Reverse Language Model
arXiv:2507.01335v4 Announce Type: replace-cross Abstract: Autoregressive language models are trained exclusively left-to-right. We explore the complementary factorization, training right-to-left at scale, and ask what reasoning patterns emerge when a model conditions on future context to predict the past. We train LEDOM, an open-source purely reverse autoregressive language model (2B/7B parameters, 435B tokens), and find it develops capabilities distinct from forward models, including abductive inference, question synthesis, and natural resolution of the reversal curse. We then explore one application of the reverse model: combining forward likelihood $P(y \mid x)$ with reverse posterior $P(x \mid y)$ through noisy channel duality. We propose Reverse Reward, which reranks forward outputs using reverse posterior estimates, and prove that bidirectional scoring penalizes hallucinated reasoning chains whose backward reconstruction degrades. Reverse Reward yields gains of up to 6.6\% on AIME 2024 and 15\% on AMC 2023 across multiple strong baselines. We release all models, code, and data here: https://github.com/Arvid-pku/LEDOM.
Abstract:Autoregressive language models are trained exclusively left-to-right. We explore the complementary factorization, training right-to-left at scale, and ask what reasoning patterns emerge when a model conditions on future context to predict the past. We train LEDOM, an open-source purely reverse autoregressive language model (2B/7B parameters, 435B tokens), and find it develops capabilities distinct from forward models, including abductive inference, question synthesis, and natural resolution of the reversal curse. We then explore one application of the reverse model: combining forward likelihood $P(y \mid x)$ with reverse posterior $P(x \mid y)$ through noisy channel duality. We propose Reverse Reward, which reranks forward outputs using reverse posterior estimates, and prove that bidirectional scoring penalizes hallucinated reasoning chains whose backward reconstruction degrades. Reverse Reward yields gains of up to 6.6\% on AIME 2024 and 15\% on AMC 2023 across multiple strong baselines. We release all models, code, and data here: this https URL.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2507.01335 [cs.CL]
(or arXiv:2507.01335v4 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 164 cuvinte
Subiecte conexe
Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptation
arXiv · 5 oct. 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · 19 mai 2026
Frontier AI Risk Report (February–March 2026)
METR · 19 mai 2026
Frontier Risk Report (February to March 2026)
METR · 19 mai 2026