MATP Framework for Verifying LLM Reasoning

This paper addresses the critical issue of logical flaws in LLM reasoning, which is crucial for the safe deployment of LLMs in high-stakes applications. The proposed MATP framework offers a novel approach by translating natural language reasoning into First-Order Logic and using automated theorem provers. This allows for a more rigorous and systematic evaluation of LLM reasoning compared to existing methods. The significant performance gains over baseline methods highlight the effectiveness of MATP and its potential to improve the trustworthiness of LLM-generated outputs.

• •MATP is a framework for verifying LLM reasoning using Multi-step Automated Theorem Proving.
• •It translates natural language reasoning into First-Order Logic and uses automated theorem provers.
• •MATP outperforms prompting-based baselines in reasoning step verification.
• •The framework reveals model-level disparities in logical coherence.