Backpropagation and KL Projections: Exact Correspondences

This paper explores the mathematical connections between backpropagation, a core algorithm in deep learning, and Kullback-Leibler (KL) divergence, a measure of the difference between probability distributions. It establishes two precise relationships, showing that backpropagation can be understood through the lens of KL projections. This provides a new perspective on how backpropagation works and potentially opens avenues for new algorithms or theoretical understanding. The focus on exact correspondences is significant, as it provides a strong mathematical foundation.