RxnBench: Evaluating LLMs on Chemical Reaction Understanding

This paper introduces RxnBench, a new benchmark to evaluate Multimodal Large Language Models (MLLMs) on their ability to understand chemical reactions from scientific literature. It highlights a significant gap in current MLLMs' ability to perform deep chemical reasoning and structural recognition, despite their proficiency in extracting explicit text. The benchmark's multi-tiered design, including Single-Figure QA and Full-Document QA, provides a rigorous evaluation framework. The findings emphasize the need for improved domain-specific visual encoders and reasoning engines to advance AI in chemistry.

• •RxnBench is a new benchmark for evaluating MLLMs on chemical reaction understanding.
• •MLLMs struggle with deep chemical logic and structural recognition.
• •Inference-time reasoning models outperform standard architectures.
• •Domain-specific visual encoders and stronger reasoning engines are needed.