aiqbench analysis
This benchmark study reveals that a simple calibrated rule-based controller often outperforms six mainstream deep reinforcement learning algorithms on cost efficiency for adaptive resource control. The research highlights that the primary bottleneck is not algorithm selection but rather issues with baseline calibration, reward engineering, and evaluation protocols under distribution shifts. Consequently, DRL's advantage appears limited to specific scenarios like bursty traffic where rule-based heuristics struggle, challenging common assumptions about RL superiority in infrastructure management.
When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
arXiv:2605.26418v3 Announce Type: replace Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.
Abstract:A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
I.2.6; I.2.8; C.2.4
arXiv:2605.26418 [cs.LG]
(or arXiv:2605.26418v3 [cs.LG] for this version)
Full text under the publisher's open licence · 214 words
Related coverage
When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
arXiv · Jun 12, 2026
Convergence in Science, Divergence in Religion: Calibrated Framing Differences Across Wikipedia's Language Editions
arXiv · Aug 25, 2026
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
arXiv · Jul 24, 2026
Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents
arXiv · Oct 7, 2026