[Model Release] Genesis-152M-Instruct: Exploring Hybrid Attention + TTT at Small Scale

This article announces the release of Genesis-152M-Instruct, a small language model designed for research purposes. It focuses on exploring the interaction of recent architectural innovations like GLA, FoX, TTT, µP, and sparsity within a constrained data environment. The key question addressed is how much architectural design can compensate for limited training data at a 150M parameter scale. The model combines several ICLR 2024-2025 ideas and includes hybrid attention, test-time training, selective activation, and µP-scaled training. While benchmarks are provided, the author emphasizes that this is not a SOTA model but rather an architectural exploration, particularly in comparison to models trained on significantly larger datasets.

• •Genesis-152M-Instruct is a small language model for architectural research.
• •It explores hybrid attention and test-time training at a small scale.
• •The model is fully open-source and available on Hugging Face.