Language and vision-language models can describe driving scenes and respond to natural-language requests. The open questions are how well they understand multi-view sensor data, whether they can carry out a driver's instructions through existing driving functions, and whether they can run within in-vehicle compute and latency limits. This work builds benchmarks and datasets for driving-scene understanding, connects language models to driving functions and to dialogue with the driver, and studies efficient inference and language-aligned scenario generation.
NuPlanQA, a multi-view question-answering benchmark and a dataset of one million question-answer pairs for driving scenes; PDB-Eval, a benchmark for describing and explaining personalized driving behavior from video; and concept-bottleneck models that explain driving control decisions with human-understandable concepts.
NuPlanQA, ICCV 2025 · PDB-Eval, IEEE IV 2025 · Echterhoff et al., WACV 2024
LaMPilot, a framework and benchmark in which language models carry out user instructions such as "overtake the car ahead" by writing programs over established driving functions; and LLM4AD, a review and benchmark of language models for autonomous driving with experiments on vehicle platforms.
Scene-Aware Conversational ADAS (SC-ADAS) uses language models, vision-to-text scene interpretation, and structured function calling for multi-turn dialogue grounded in the driving scene, and executes ADAS actions only after the driver confirms them.
Video token sparsification removes redundant visual tokens across consecutive frames, raising multimodal-LLM inference throughput by up to 33% and cutting memory use by up to 28% in driving visual question answering.
SIMSplat, language-aligned 4D Gaussian splatting for generating driving scenarios; and NIVA, a probabilistic closed-loop multi-agent traffic simulator evaluated on the Waymo Open Motion Dataset.
A curated selection. Citations above link to each paper; see the full publication list for all work.
S. Park, C. Cui, Y. Ma, A. Moradipari, R. Gupta, K. Han, and Z. Wang
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Y. Ma, C. Cui, X. Cao, W. Ye, P. Liu, J. Lu, A. Abdelraouf, R. Gupta, K. Han, A. Bera, J. Rehg, and Z. Wang
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA
C. Cui, Y. Ma, S. Park, Z. Yang, Y. Zhou, P. Liu, J. Lu, J. Peng, J. Zhang, R. Zhang, L. Li, Y. Chen, J. Panchal, A. Abdelraouf, R. Gupta, K. Han, and Z. Wang
Proceedings of the IEEE, pp. 1–36
K. Han, Y. Chen, R. Gupta, and O. Altintas
2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC)
J. Wu, J. Echterhoff, K. Han, A. Abdelraouf, R. Gupta, and J. McAuley
2025 IEEE Intelligent Vehicles Symposium (IV), Cluj-Napoca, Romania
S. Park, A. Lee, J. Lu, C. Cui, L. Jiang, R. Gupta, K. Han, A. Moradipari, and Z. Wang
European Conference on Computer Vision (ECCV)