50M param PGN-only transformer plays coherent chess without search: Is small-LLM generalization is underrated?
Analysis
Key Takeaways
- •Small, domain-trained LLMs can show sharp in-distribution generalization.
- •The model plays coherent chess using only PGN data.
- •The model samples a move distribution instead of crunching Stockfish lines.
- •The model is 'Stockfish-trained' to imitate Stockfish's choices.
- •Temperature settings affect model behavior.
“The article highlights the model's ability to sample a move distribution instead of crunching Stockfish lines, and its 'Stockfish-trained' nature, meaning it imitates Stockfish's choices without using the engine itself. It also mentions temperature sweet-spots for different model styles.”