Eliciting Behaviors in Multi-Turn Conversations

This paper addresses the critical problem of evaluating large language models (LLMs) in multi-turn conversational settings. It extends existing behavior elicitation techniques, which are primarily designed for single-turn scenarios, to the more complex multi-turn context. The paper's contribution lies in its analytical framework for categorizing elicitation methods, the introduction of a generalized multi-turn formulation for online methods, and the empirical evaluation of these methods on generating multi-turn test cases. The findings highlight the effectiveness of online methods in discovering behavior-eliciting inputs, especially compared to static methods, and emphasize the need for dynamic benchmarks in LLM evaluation.