aiqbench analysis
A rigorous benchmark reveals that properly calibrated rule-based autoscalers consistently outperform six mainstream deep reinforcement learning algorithms on cost across diverse workloads, challenging the assumption that DRL is inherently superior for adaptive resource control. The study identifies baseline calibration, reward engineering, and realistic evaluation protocols as the primary bottlenecks rather than algorithm selection, noting that discrete-action methods significantly reduce constraint violations compared to continuous-action counterparts. While RL agents retain an edge on specific bursty traffic patterns, the findings suggest that practical deployment should prioritize robust baseline design over complex model training.
When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
arXiv:2605.26418v2 Announce Type: replace Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.
Title:When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
Abstract:A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
I.2.6; I.2.8; C.2.4
arXiv:2605.26418 [cs.LG]
(or arXiv:2605.26418v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.26418 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 309 words
Related coverage
Convergence in Science, Divergence in Religion: Calibrated Framing Differences Across Wikipedia's Language Editions
arXiv · Aug 25, 2026
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
arXiv · Jul 24, 2026
When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
arXiv · Jun 16, 2026
Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents
arXiv · Oct 7, 2026