Accelerating Visual Autoregressive Models with Adaptive Draft Trees

This paper addresses the slow inference speed of autoregressive (AR) image models, which is a significant bottleneck. It proposes a novel method, Adjacency-Adaptive Dynamical Draft Trees (ADT-Tree), to accelerate inference by dynamically adjusting the draft tree structure based on the complexity of different image regions. This is a crucial improvement over existing speculative decoding methods that struggle with the spatially varying prediction difficulty in visual AR models. The results show significant speedups on benchmark datasets.

• •Addresses the slow inference problem of autoregressive image models.
• •Proposes Adjacency-Adaptive Dynamical Draft Trees (ADT-Tree) for faster inference.
• •ADT-Tree dynamically adjusts draft tree structure based on image region complexity.
• •Achieves significant speedups on benchmark datasets.
• •Integrates with relaxed sampling methods for further acceleration.