mHC: Stabilizing and Scaling Hyper-Connections with Manifold Constraints

This paper addresses the instability and scalability issues of Hyper-Connections (HC), a recent advancement in neural network architecture. HC, while improving performance, loses the identity mapping property of residual connections, leading to training difficulties. mHC proposes a solution by projecting the HC space onto a manifold, restoring the identity mapping and improving efficiency. This is significant because it offers a practical way to improve and scale HC-based models, potentially impacting the design of future foundational models.