M$^3$KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
Analysis
This paper introduces M$^3$KG-RAG, a novel approach to Retrieval-Augmented Generation (RAG) that leverages multi-hop multimodal knowledge graphs (MMKGs) to enhance the reasoning and grounding capabilities of multimodal large language models (MLLMs). The key innovations include a multi-agent pipeline for constructing multi-hop MMKGs and a GRASP (Grounded Retrieval And Selective Pruning) mechanism for precise entity grounding and redundant context pruning. The paper addresses limitations in existing multimodal RAG systems, particularly in modality coverage, multi-hop connectivity, and the filtering of irrelevant knowledge. The experimental results demonstrate significant improvements in MLLMs' performance across various multimodal benchmarks, suggesting the effectiveness of the proposed approach in enhancing multimodal reasoning and grounding.
Key Takeaways
“To address these limitations, we propose M$^3$KG-RAG, a Multi-hop Multimodal Knowledge Graph-enhanced RAG that retrieves query-aligned audio-visual knowledge from MMKGs, improving reasoning depth and answer faithfulness in MLLMs.”