Risk-Aware Alignment for Safer Language Models

This paper addresses the critical issue of safety in fine-tuning language models. It moves beyond risk-neutral approaches by introducing a novel method, Risk-aware Stepwise Alignment (RSA), that explicitly considers and mitigates risks during policy optimization. This is particularly important for preventing harmful behaviors, especially those with low probability but high impact. The use of nested risk measures and stepwise alignment is a key innovation, offering both control over model shift and suppression of dangerous outputs. The theoretical analysis and experimental validation further strengthen the paper's contribution.