- License
- Unknown
- :sunglasses: Awesome
- Projects
Specs
- :sunglasses: AwesomeProjects
README
English | 简体中文
Are VLMs Ready for Autonomous Driving?An Empirical Study from the Reliability, Data, and Metric Perspectives
8
About
1
-
This work introduces :blue_car: DriveBench , a benchmark dataset designed to evaluate VLM reliability across 17 settings (clean, corrupted, and text-only inputs), encompassing 19,200 frames , 20,498 question-answer pairs , three question types , four mainstream driving tasks , and a total of 12 popular VLMs . Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding, especially under degraded or missing visual inputs. This behavior, concealed by dataset imbalances and insufficient evaluation metrics , poses significant risks in safety-critical scenarios like autonomous driving.
:books: Citation
1
-
If you find this work helpful for your research, please kindly consider citing our papers:
:memo: Updates
19
-
Benchmark Comparison
-
Installation
-
Data Preparation
-
Getting Started
-
Benchmark Results
-
Benchmark Configuration
-
Benchmark Study
-
Robustness Analysis
-
License
-
Acknowledgments
Benchmark Configuration
1
-
<details open> <summary> <b>Commercial VLMs</b></summary>
Benchmark Study
1
-
<table> <thead> <tr> <th>Model</th> <th>Size</th> <th>Type</th> <th>
Robustness Analysis
1
-
<table> <thead> <tr> <th rowspan="2">Model</th> <th rowspan="2">Size</th> <th rowspan="2">Type</th> <th colspan="3">
Qualitative Comparisons
1
-
Examples of different VLM responses under the Frame Lost condition. We observe that GPT-4o responses with visible objects while LLaVA-NeXT and DriveLM tend to hallucinate objects that cannot be seen from the provided images.