Understanding Fast Hyperparameter Transfer in Deep Learning

This paper addresses the critical problem of hyperparameter optimization in large-scale deep learning. It investigates the phenomenon of fast hyperparameter transfer, where optimal hyperparameters found on smaller models can be effectively transferred to larger models. The paper provides a theoretical framework for understanding this transfer, connecting it to computational efficiency. It also explores the mechanisms behind fast transfer, particularly in the context of Maximal Update Parameterization ($μ$P), and provides empirical evidence to support its hypotheses. The work is significant because it offers insights into how to efficiently optimize large models, a key challenge in modern deep learning.