analiza aiqbench
The SCOPE framework establishes a critical benchmark for edge-deployed language agents by validating that sufficiently capable small language models (SLMs) effectively reduce hallucinations while shifting the primary performance bottleneck to perception. The study demonstrates that mixture-of-experts architectures can match dense models in latency and memory efficiency, with quantization offering further gains for real-time PTZ control. This work provides a practical, sim-to-real validated design point for building reliable, closed-loop visual agents constrained by edge compute resources.
SCOPE: Real-Time Natural Language Camera Agent at the Edge
arXiv:2606.02951v1 Announce Type: cross Abstract: Deploying language-driven agents in robotics requires evaluations that reflect real-world task demands: natural-language instructions with reproducible outcomes. Such agents must connect language models to callable perception and control tools, and be assessed using deployment-critical metrics including latency, accuracy, and error modes. We present SCOPE (Simulation and Camera Operations for Perception and Evaluation), a modular agent for natural-language, open-vocabulary pan-tilt-zoom (PTZ) camera control and visual scene understanding, designed explicitly for edge deployment. SCOPE operates both in a Blender-based simulation environment and on a physical PTZ camera, executing all perception, planning, and control locally at the deployment site using edge-accessible compute. We release a 536-task benchmark spanning QA, single- and multi-step commands, counting, spatial reasoning, descriptions, and optical character recognition in a Blender-based simulation environment that exposes realistic PTZ control affordances. Execution traces are combined with an LM-as-Judge to evaluate latency, accuracy, and error modes. We evaluate 19 planner-perception model combinations pairing Qwen3 small language models (SLMs) with Moondream and Qwen vision-language models (VLMs). Stronger SLMs substantially reduce hallucinations and improve tool routing, leading to more reliable closed-loop behavior. Once a sufficiently capable SLM is used, perception becomes the dominant performance bottleneck. Mixture-of-Experts models on both the planning and perception side consistently match or exceed dense alternatives at latencies and memory footprints comparable to much smaller networks. Quantization provides additional efficiency gains with minimal accuracy degradation, identifying a practical, sim-to-real validated design point for real-time, edge-feasible language-driven PTZ control.
Abstract:Deploying language-driven agents in robotics requires evaluations that reflect real-world task demands: natural-language instructions with reproducible outcomes. Such agents must connect language models to callable perception and control tools, and be assessed using deployment-critical metrics including latency, accuracy, and error modes. We present SCOPE (Simulation and Camera Operations for Perception and Evaluation), a modular agent for natural-language, open-vocabulary pan-tilt-zoom (PTZ) camera control and visual scene understanding, designed explicitly for edge deployment. SCOPE operates both in a Blender-based simulation environment and on a physical PTZ camera, executing all perception, planning, and control locally at the deployment site using edge-accessible compute. We release a 536-task benchmark spanning QA, single- and multi-step commands, counting, spatial reasoning, descriptions, and optical character recognition in a Blender-based simulation environment that exposes realistic PTZ control affordances. Execution traces are combined with an LM-as-Judge to evaluate latency, accuracy, and error modes. We evaluate 19 planner-perception model combinations pairing Qwen3 small language models (SLMs) with Moondream and Qwen vision-language models (VLMs). Stronger SLMs substantially reduce hallucinations and improve tool routing, leading to more reliable closed-loop behavior. Once a sufficiently capable SLM is used, perception becomes the dominant performance bottleneck. Mixture-of-Experts models on both the planning and perception side consistently match or exceed dense alternatives at latencies and memory footprints comparable to much smaller networks. Quantization provides additional efficiency gains with minimal accuracy degradation, identifying a practical, sim-to-real validated design point for real-time, edge-feasible language-driven PTZ control.
9 pages, 4 figures, 6 tables. Accepted at HRI '26 (21st ACM/IEEE International Conference on Human-Robot Interaction), Edinburgh, Scotland, March 16--19, 2026. Code: this https URL
Robotics (cs.RO); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Human-Computer Interaction (cs.HC)
I.2.9; I.2.10; I.2.7; I.2.11
arXiv:2606.02951 [cs.RO]
(or arXiv:2606.02951v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.02951 arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3757279.3785641 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 306 cuvinte
Subiecte conexe
Build a Multi-Camera 3D Tracking Application with NVIDIA DeepStream 9.1 Skills
NVIDIA · 15 iul. 2026
Record, train, and deploy from one place with Strands Agents, LeRobot, and Hugging Face Storage Buckets
Hugging Face · 13 aug. 2026
NVIDIA and Japan Bring Full-Stack AI and Robotics to Every Industry
NVIDIA · 15 iul. 2026
How to Build In-Vehicle AI Agents with NVIDIA: From Cloud to Car
NVIDIA · 5 mai 2026