NOMA: Neural Networks That Reallocate Themselves During Training
Published:Dec 26, 2025 13:40
•1 min read
•r/MachineLearning
Analysis
This article discusses NOMA, a novel systems language and compiler designed for neural networks. Its key innovation lies in implementing reverse-mode autodiff as a compiler pass, enabling dynamic network topology changes during training without the overhead of rebuilding model objects. This approach allows for more flexible and efficient training, particularly in scenarios involving dynamic capacity adjustment, pruning, or neuroevolution. The ability to preserve optimizer state across growth events is a significant advantage. The author highlights the contrast with typical Python frameworks like PyTorch and TensorFlow, where such changes require significant code restructuring. The provided example demonstrates the potential for creating more adaptable and efficient neural network training pipelines.
Key Takeaways
Reference
“In NOMA, a network is treated as a managed memory buffer. Growing capacity is a language primitive.”