50M param PGN-only transformer plays coherent chess without search: Is small-LLM generalization is underrated?
Analysis
This article discusses a 50 million parameter transformer model trained on PGN data that plays chess without search. The model demonstrates surprisingly legal and coherent play, even achieving a checkmate in a rare number of moves. It highlights the potential of small, domain-specific LLMs for in-distribution generalization compared to larger, general models. The article provides links to a write-up, live demo, Hugging Face models, and the original blog/paper.
Key Takeaways
- •Small, domain-trained LLMs can show sharp in-distribution generalization.
- •The model plays coherent chess using only PGN data.
- •The model samples a move distribution instead of crunching Stockfish lines.
- •The model is 'Stockfish-trained' to imitate Stockfish's choices.
- •Temperature settings affect model behavior.
Reference / Citation
View Original"The article highlights the model's ability to sample a move distribution instead of crunching Stockfish lines, and its 'Stockfish-trained' nature, meaning it imitates Stockfish's choices without using the engine itself. It also mentions temperature sweet-spots for different model styles."