Jailbreak Attacks vs. Content Safety Filters: LLM Safety Evaluation

This paper addresses a critical gap in LLM safety research by evaluating jailbreak attacks within the context of the entire deployment pipeline, including content moderation filters. It moves beyond simply testing the models themselves and assesses the practical effectiveness of attacks in a real-world scenario. The findings are significant because they suggest that existing jailbreak success rates might be overestimated due to the presence of safety filters. The paper highlights the importance of considering the full system, not just the LLM, when evaluating safety.