Search:
Match:
3 results
Research#llm📝 BlogAnalyzed: Dec 27, 2025 22:31

[D] NOMA update: reproducible self-growing XOR benchmark (shared init, N=10) + optimizer-state “preserve vs reset” ablation

Published:Dec 27, 2025 22:14
1 min read
r/MachineLearning

Analysis

This post details an update on NOMA, a system language and compiler focused on implementing reverse-mode autodiff as a compiler pass. The key addition is a reproducible benchmark for a "self-growing XOR" problem. This benchmark allows for controlled comparisons between different implementations, focusing on the impact of preserving or resetting optimizer state during parameter growth. The use of shared initial weights and a fixed growth trigger enhances reproducibility. While XOR is a simple problem, the focus is on validating the methodology for growth events and assessing the effect of optimizer state preservation, rather than achieving real-world speed.
Reference

“The goal here is methodology validation: making the growth event comparable, checking correctness parity, and measuring whether preserving optimizer state across resizing has a visible effect.”

Permalinkr/MachineLearning
Research#llm📝 BlogAnalyzed: Dec 26, 2025 13:44

NOMA: Neural Networks That Reallocate Themselves During Training

Published:Dec 26, 2025 13:40
1 min read
r/MachineLearning

Analysis

This article discusses NOMA, a novel systems language and compiler designed for neural networks. Its key innovation lies in implementing reverse-mode autodiff as a compiler pass, enabling dynamic network topology changes during training without the overhead of rebuilding model objects. This approach allows for more flexible and efficient training, particularly in scenarios involving dynamic capacity adjustment, pruning, or neuroevolution. The ability to preserve optimizer state across growth events is a significant advantage. The author highlights the contrast with typical Python frameworks like PyTorch and TensorFlow, where such changes require significant code restructuring. The provided example demonstrates the potential for creating more adaptable and efficient neural network training pipelines.
Reference

“In NOMA, a network is treated as a managed memory buffer. Growing capacity is a language primitive.”

Permalinkr/MachineLearning
Research#llm📝 BlogAnalyzed: Dec 26, 2025 16:47

Calculus on Computational Graphs: Backpropagation

Published:Aug 31, 2015 00:00
1 min read
Colah

Analysis

This article provides a clear and concise explanation of backpropagation, emphasizing its crucial role in making deep learning computationally feasible. It highlights the algorithm's efficiency compared to naive implementations and its broader applicability beyond deep learning, such as in weather forecasting and numerical stability analysis. The article also points out that backpropagation, or reverse-mode differentiation, has been independently discovered in various fields. The author effectively conveys the fundamental nature of backpropagation as a technique for rapid derivative calculation, making it a valuable tool in diverse numerical computing scenarios. The article's accessibility makes it suitable for readers with varying levels of technical expertise.
Reference

“Backpropagation is the key algorithm that makes training deep models computationally tractable.”

PermalinkColah