DarkEQA: Benchmarking VLMs for Low-Light Embodied Question Answering
Analysis
This paper addresses a critical gap in the evaluation of Vision-Language Models (VLMs) for embodied agents. Existing benchmarks often overlook the performance of VLMs under low-light conditions, which are crucial for real-world, 24/7 operation. DarkEQA provides a novel benchmark to assess VLM robustness in these challenging environments, focusing on perceptual primitives and using a physically-realistic simulation of low-light degradation. This allows for a more accurate understanding of VLM limitations and potential improvements.
Key Takeaways
- •Introduces DarkEQA, a new benchmark for evaluating VLMs in low-light embodied question answering.
- •Employs a physically-realistic simulation of low-light conditions.
- •Enables attributable robustness analysis by isolating the perception bottleneck.
- •Evaluates state-of-the-art VLMs and LLIE models, revealing their limitations.
Reference / Citation
View Original"DarkEQA isolates the perception bottleneck by evaluating question answering from egocentric observations under controlled degradations, enabling attributable robustness analysis."