GRAPE: A New Positional Encoding for Long-Context Models

This paper introduces GRAPE (Group Representational Position Encoding), a novel framework for positional encoding in long-context models, improving upon existing methods like RoPE and ALiBi. The research explores both multiplicative and additive approaches, offering a flexible design space for capturing positional information, and aims to enhance the performance of long-context models.