Import AI 479: Spacewar; low-latency tasks versus LLMs; Genie 4

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. LLMs are very bad at low-latency tasks:…Realtime-Bench says it’ll be a while till AI systems can play games or do online auctions for you…Researchers have built Realtime-Bench, “a suite of carefully curated dynamic tasks where delayed actions directly affect success”. The test helps evaluate how well modern, general-purpose AI systems can perform tasks where decisions need to be made in real-time. The benchmark consists of 50 interactive game and web tasks, split across 30 web tasks in four categories (travel, e-commerce, service, and security), and 20 game tasks (arcade, puzzle, strategy, and simulation). Who did the research: The benchmark was built by people at the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks. What it measures: The test is designed to probe for three qualities: temporal state awareness (”RT-Perceive”), where models can use observations to track changes in the world; adaptive deliberation (”RT-Deliberate”), where models alter their reasoning length and depth according to available time; and time-aware action selection (”RT-Act”) where models need…