RAG Risks: Why Retrieval-Augmented LLMs are Not Safer with Sebastian Gehrmann

This article discusses the safety risks associated with Retrieval-Augmented Generation (RAG) systems, particularly in high-stakes domains like financial services. It highlights that RAG, despite expectations, can degrade model safety, leading to unsafe outputs. The discussion covers evaluation methods for these risks, potential causes for the counterintuitive behavior, and a domain-specific safety taxonomy for the financial industry. The article also emphasizes the importance of governance, regulatory frameworks, prompt engineering, and mitigation strategies to improve AI safety within specialized domains. The interview with Sebastian Gehrmann, head of responsible AI at Bloomberg, provides valuable insights.