Learning Rate Decay: A Hidden Bottleneck in LLM Curriculum Pretraining

This ArXiv paper critically examines the detrimental effects of learning rate decay in curriculum-based pretraining of Large Language Models (LLMs). The research likely highlights how traditional decay schedules can lead to the suboptimal utilization of high-quality training data early in the process.

• •Learning rate decay in curriculum learning can lead to inefficient use of high-quality data.
• •The research suggests that alternative learning rate schedules might improve performance.
• •This work has implications for optimizing the pretraining process of LLMs.