GEM Activations: The Smooth New Functions Outperforming GELU in Transformers and CNNs

This research introduces an exciting advancement in neural network architecture by proposing the Geometric Monomial (GEM) family of activation functions. By achieving ReLU-like performance with purely rational arithmetic and smooth gradients, GEM effectively solves optimization hurdles in deep architectures. Impressively, the SE-GEM and E-GEM variants have successfully surpassed the industry-standard GELU function on major benchmarks like CIFAR-10 and GPT-2, marking a significant milestone for model optimization.