Measuring and Steering LLM Computation with Multiple Token Divergence
Analysis
This paper introduces a novel method, Multiple Token Divergence (MTD), to measure and control the computational effort of language models during in-context learning. It addresses the limitations of existing methods by providing a non-invasive and stable metric. The proposed Divergence Steering method offers a way to influence the complexity of generated text. The paper's significance lies in its potential to improve the understanding and control of LLM behavior, particularly in complex reasoning tasks.
Key Takeaways
- •Proposes Multiple Token Divergence (MTD) as a new metric for measuring computational effort in LLMs.
- •Introduces Divergence Steering, a method to control the computational character of generated text.
- •MTD is shown to correlate with problem difficulty and accuracy in mathematical reasoning tasks.
- •MTD is a practical and lightweight tool for analyzing and steering LLM dynamics.
Reference
“MTD is more effective than prior methods at distinguishing complex tasks from simple ones. Lower MTD is associated with more accurate reasoning.”