aiqbench analysis
This research addresses the production challenges of technical-service LLM agents by introducing a framework that jointly models latent decision logic, response diversity, and reward costs. By combining planning-aware trajectory modeling with hybrid reward mechanisms, the approach significantly improves adoption rates while reducing evaluation expenses in noisy environments. The results demonstrate that adapting base models like Qwen3-4B for specific industrial workflows requires moving beyond standard post-training to account for hidden reasoning structures and data noise.
Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modeling
arXiv:2603.18074v2 Announce Type: replace Abstract: Technical-service LLM agents are entering production workflows, where value depends on whether engineers adopt generated replies. Service tickets hide decision logic, contain noisy single-reference responses, and make reward evaluation costly, making standard post-training brittle. Existing post-training and LLM-as-a-Judge approaches improve grounding or feedback, but do not jointly model latent decision logic, response diversity, and reward cost. We address this gap by coupling latent logic augmentation, robust noise reduction, and hybrid reward modeling. The framework augments supervised fine-tuning data with Planning-Aware Trajectory Modeling and Reasoning Augmentation, builds dual-filtered Multiple Ground Truths, and trains the policy with a hybrid reward that combines a Reranker with an LLM-as-a-Judge. On real Cloud technical-service tasks, the adapted Qwen3-4B achieves the highest Multi-ECS (0.441), lower reward cost, and the highest production adoption rate (46.63%).
Abstract:Technical-service LLM agents are entering production workflows, where value depends on whether engineers adopt generated replies. Service tickets hide decision logic, contain noisy single-reference responses, and make reward evaluation costly, making standard post-training brittle. Existing post-training and LLM-as-a-Judge approaches improve grounding or feedback, but do not jointly model latent decision logic, response diversity, and reward cost. We address this gap by coupling latent logic augmentation, robust noise reduction, and hybrid reward modeling. The framework augments supervised fine-tuning data with Planning-Aware Trajectory Modeling and Reasoning Augmentation, builds dual-filtered Multiple Ground Truths, and trains the policy with a hybrid reward that combines a Reranker with an LLM-as-a-Judge. On real Cloud technical-service tasks, the adapted Qwen3-4B achieves the highest Multi-ECS (0.441), lower reward cost, and the highest production adoption rate (46.63%).
36 pages, 6 figures, 14 tables. Camera-ready version accepted to the EMNLP 2026 Industry Track. Title, author order and metadata, experiments, analysis, references, and appendices have been updated; the set of authors is unchanged
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Applications (stat.AP)
arXiv:2603.18074 [cs.LG]
(or arXiv:2603.18074v2 [cs.LG] for this version)
Full text under the publisher's open licence · 181 words
