analiza aiqbench
This research introduces ER-JEPA, a method that enhances Large Language Models by integrating episodic replay into Joint-Embedding Predictive Architectures to improve representation stability and alignment. By leveraging stored training pairs for additional supervision, the approach demonstrates consistent performance gains across multiple benchmarks without relying on external knowledge bases. This advancement addresses critical limitations in current LLMs regarding abstract semantics and perception through a more robust learning mechanism.
ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models
arXiv:2609.36952v1 Announce Type: new Abstract: Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA mitigates this by aligning different views of the same underlying knowledge via a joint-embedding predictive architecture (JEPA). However, strong alignment does not necessarily lead to accurate, stable predictions. To address this, we propose ER-JEPA, which adds an episodic replay path to LLM-JEPA. ER-JEPA stores training pairs in a memory. At each step, it stores and retrieves relevant data to provide additional supervision. This enables learning from both the current batch and stored training pairs, providing additional supervision for token prediction and representation alignment. Experiments across multiple datasets (NL-RX, GSM8K, Spider, and NQ-Open) demonstrate that ER-JEPA consistently outperforms LLM-JEPA.
Abstract:Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA mitigates this by aligning different views of the same underlying knowledge via a joint-embedding predictive architecture (JEPA). However, strong alignment does not necessarily lead to accurate, stable predictions. To address this, we propose ER-JEPA, which adds an episodic replay path to LLM-JEPA. ER-JEPA stores training pairs in a memory. At each step, it stores and retrieves relevant data to provide additional supervision. This enables learning from both the current batch and stored training pairs, providing additional supervision for token prediction and representation alignment. Experiments across multiple datasets (NL-RX, GSM8K, Spider, and NQ-Open) demonstrate that ER-JEPA consistently outperforms LLM-JEPA.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.36952 [cs.CL]
(or arXiv:2609.36952v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.36952 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 215 cuvinte
Subiecte conexe
HP-JEPA: Hierarchical Partitioning for Multi-Resolution Graph Joint-Embedding Predictive Learning
arXiv · 4 aug. 2026
MA-JEPA: Joint-Embedding World Models for Multi-Agent Reinforcement Learning
arXiv · 30 sept. 2026
4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting
arXiv · 24 sept. 2026
CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
arXiv · 7 oct. 2026