LLMs Learn Like a Swiss Army Knife: Context Structure Reveals Dynamic Strategies

This research explores how Large Language Models (LLMs) adapt their representational geometry during in-context learning. The study uncovers a fascinating dichotomy, showing LLMs dynamically choosing between different strategies depending on task structure, leading to improved prediction performance.

• •LLMs' representational straightness increases in continual prediction settings.
• •In structured prediction, straightening only occurs in phases with explicit structure.
• •The study suggests LLMs employ different strategies based on the task.