Human-Centered Framework for Evaluating AI Agents in Software Engineering

This paper addresses a critical gap in AI evaluation by shifting the focus from code correctness to collaborative intelligence. It recognizes that current benchmarks are insufficient for evaluating AI agents that act as partners to software engineers. The paper's contributions, including a taxonomy of desirable agent behaviors and the Context-Adaptive Behavior (CAB) Framework, provide a more nuanced and human-centered approach to evaluating AI agent performance in a software engineering context. This is important because it moves the field towards evaluating the effectiveness of AI agents in real-world collaborative scenarios, rather than just their ability to generate correct code.