MoR: Dynamic Mixed-Precision Training

This paper introduces Mixture-of-Representations (MoR), a novel framework for mixed-precision training. It dynamically selects between different numerical representations (FP8 and BF16) at the tensor and sub-tensor level based on the tensor's properties. This approach aims to improve the robustness and efficiency of low-precision training, potentially enabling the use of even lower precision formats like NVFP4. The key contribution is the dynamic, property-aware quantization strategy.

• •Proposes MoR, a dynamic mixed-precision training framework.
• •Dynamically selects between FP8 and BF16 representations.
• •Achieves state-of-the-art results with high FP8 usage.
• •Aims to improve robustness and enable lower precision formats.