Apple Neural Engine Secret Unlocked: Powering Tiny LLMs!
Analysis
This exciting development unveils a new way to train smaller models using Apple's Neural Engine (ANE). The ability to reverse engineer the ANE and create a specialized training pipeline is a fantastic leap forward. The power efficiency demonstrated is truly remarkable, opening doors for energy-conscious AI development.
Key Takeaways
- •Researchers have reverse-engineered Apple's Neural Engine (ANE) to train a small Generative AI model.
- •The project bypasses the standard CoreML framework for direct ANE access, creating a bespoke training pipeline.
- •ANE demonstrates incredible power efficiency, significantly outperforming Metal GPU and even leading GPUs like the H100 in terms of TFLOPS/watt.
Reference / Citation
View Original"Peak compute on ANE only consumes 2.8 W which at 19 tflops becomes 6.6 tflops/watt. Insane!"