ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model

Abstract:Comprehensive traffic scene understanding is a foundational capability for Intelligent Transportation Systems (ITS) underpinning applications such as traffic simulation. While VisionLanguage Models (VLMs) have demonstrated strong reasoning potential, their application to Bird's-Eye View (BEV) maps in traffic contexts remains limited by narrow task definitions and scarce annotated data. We introduce ChatBEV-QA, a large-scale BEV VQA benchmark of 137K+ QA pairs, designed to evaluate global scene understanding, vehicle-lane interactions, and vehiclevehicle interactions within complex traffic environments. Building on this, we fine-tune ChatBEV, a specialized VLM that accurately interprets diverse scene understanding queries from BEV maps. To demonstrate downstream utility in ITS applications, we integrate ChatBEV into a language-guided traffic simulation framework. Its global understanding and navigation reasoning provide crucial context-aware guidance, reducing trajectory displacement error by up to 20.9% and scenario collision rates by up to 37.9% over text-only baselines.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

arXiv:2503.13938 [cs.CV]

(or arXiv:2503.13938v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2503.13938 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.