analiza aiqbench
This research resolves a critical bottleneck in deep reinforcement learning by introducing Stream-X, a framework that enables models to learn directly from continuous data streams without relying on memory buffers. The approach successfully overcomes the "stream barrier," allowing algorithms to maintain performance comparable to batch-based methods across diverse tasks ranging from Atari games to real-world robot navigation. This advancement is significant for deploying AI agents in dynamic environments where storing historical experience is impractical due to latency or resource constraints.
Streaming Deep Reinforcement Learning Finally Works
arXiv:2410.14606v3 Announce Type: replace Abstract: Learning from a stream of experience as it arrives, also known as streaming learning, is a core part of natural learning. However, reliable streaming learning has remained a persistent challenge in modern deep reinforcement learning (RL). Instead, most deep RL algorithms learn from old experience by storing past interactions in a buffer. We show that both classical streaming RL, such as Q-learning and actor-critic, when used with deep neural networks, and batch deep RL, such as PPO, SAC, and DQN, when adapted to the streaming setting, often fail to learn. Across 58 Atari games and 50 continuous-control tasks, we find that these methods, in aggregate, perform close to random policies despite extensive task-specific hyperparameter searches. We call this pattern stream barrier. Here, we introduce Stream-X, a shared recipe for streaming deep RL algorithms that combines signal normalization, representation stabilization, and controlled parameter updates. By applying Stream-X to several base streaming RL algorithms, we provide the first family of deep RL algorithms to overcome the stream barrier. Using one prescribed hyperparameter configuration per algorithm across tasks, Stream-X substantially improves aggregate performance, often on par with batch RL algorithms. Beyond these benchmarks, we demonstrate learning with Stream-X algorithms under nonstationarity and resource constraints. Stream-AC, one of the Stream-X algorithms, repeatedly recovers performance across alternating floor-friction regimes in simulation, outperforming the evaluated PPO and SAC baselines. It also learns a heading tracking task on a robot using proprioceptive and visual features from the on-board camera in a naturally changing laboratory environment. Stream-Q learns a Pong game from pixels directly on an ESP32-S3 microcontroller, a device with limited compute and memory.
Abstract:Learning from a stream of experience as it arrives, also known as streaming learning, is a core part of natural learning. However, reliable streaming learning has remained a persistent challenge in modern deep reinforcement learning (RL). Instead, most deep RL algorithms learn from old experience by storing past interactions in a buffer. We show that both classical streaming RL, such as Q-learning and actor-critic, when used with deep neural networks, and batch deep RL, such as PPO, SAC, and DQN, when adapted to the streaming setting, often fail to learn. Across 58 Atari games and 50 continuous-control tasks, we find that these methods, in aggregate, perform close to random policies despite extensive task-specific hyperparameter searches. We call this pattern stream barrier. Here, we introduce Stream-X, a shared recipe for streaming deep RL algorithms that combines signal normalization, representation stabilization, and controlled parameter updates. By applying Stream-X to several base streaming RL algorithms, we provide the first family of deep RL algorithms to overcome the stream barrier. Using one prescribed hyperparameter configuration per algorithm across tasks, Stream-X substantially improves aggregate performance, often on par with batch RL algorithms. Beyond these benchmarks, we demonstrate learning with Stream-X algorithms under nonstationarity and resource constraints. Stream-AC, one of the Stream-X algorithms, repeatedly recovers performance across alternating floor-friction regimes in simulation, outperforming the evaluated PPO and SAC baselines. It also learns a heading tracking task on a robot using proprioceptive and visual features from the on-board camera in a naturally changing laboratory environment. Stream-Q learns a Pong game from pixels directly on an ESP32-S3 microcontroller, a device with limited compute and memory.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2410.14606 [cs.LG]
(or arXiv:2410.14606v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2410.14606 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 362 cuvinte
Subiecte conexe
Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark
AMD · 18 aug. 2026
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics
Latent Space · 13 oct. 2026
NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI
NVIDIA · 21 mai 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · 4 sept. 2026