GeoBench: A Hierarchical Benchmark for Geometric Problem Solving
Analysis
This paper introduces GeoBench, a new benchmark designed to address limitations in existing evaluations of vision-language models (VLMs) for geometric reasoning. It focuses on hierarchical evaluation, moving beyond simple answer accuracy to assess reasoning processes. The benchmark's design, including formally verified tasks and a focus on different reasoning levels, is a significant contribution. The findings regarding sub-goal decomposition, irrelevant premise filtering, and the unexpected impact of Chain-of-Thought prompting provide valuable insights for future research in this area.
Key Takeaways
- •GeoBench provides a more comprehensive and nuanced evaluation of VLMs for geometric problem-solving.
- •The benchmark emphasizes reasoning processes over just final answers.
- •Sub-goal decomposition and irrelevant premise filtering are crucial for accuracy.
- •Chain-of-Thought prompting's impact can be task-dependent and potentially detrimental.
“Key findings demonstrate that sub-goal decomposition and irrelevant premise filtering critically influence final problem-solving accuracy, whereas Chain-of-Thought prompting unexpectedly degrades performance in some tasks.”