HiSciBench: A Hierarchical Benchmark for Scientific Intelligence
Analysis
This paper introduces HiSciBench, a novel benchmark designed to evaluate large language models (LLMs) and multimodal models on scientific reasoning. It addresses the limitations of existing benchmarks by providing a hierarchical and multi-disciplinary framework that mirrors the complete scientific workflow, from basic literacy to scientific discovery. The benchmark's comprehensive nature, including multimodal inputs and cross-lingual evaluation, allows for a detailed diagnosis of model capabilities across different stages of scientific reasoning. The evaluation of leading models reveals significant performance gaps, highlighting the challenges in achieving true scientific intelligence and providing actionable insights for future model development. The public release of the benchmark will facilitate further research in this area.
Key Takeaways
- •HiSciBench is a new hierarchical benchmark for evaluating scientific intelligence in LLMs and multimodal models.
- •It covers a complete scientific workflow from literacy to discovery.
- •The benchmark supports multimodal inputs and cross-lingual evaluation.
- •Evaluations reveal significant performance gaps in current models.
- •The benchmark will be publicly released to facilitate future research.
“While models achieve up to 69% accuracy on basic literacy tasks, performance declines sharply to 25% on discovery-level challenges.”