Optimizing Distributed LLM Inference Resource Allocation

This paper addresses the critical problem of optimizing resource allocation for distributed inference of Large Language Models (LLMs). It's significant because LLMs are computationally expensive, and distributing the workload across geographically diverse servers is a promising approach to reduce costs and improve accessibility. The paper provides a systematic study, performance models, optimization algorithms (including a mixed integer linear programming approach), and a CPU-only simulator. This work is important for making LLMs more practical and accessible.

• •Addresses the resource allocation problem for distributed LLM inference.
• •Proposes performance models for predicting inference performance.
• •Formulates the optimization problem as mixed integer linear programming.
• •Develops a CPU-only simulator for performance evaluation.
• •Demonstrates improved inference time compared to state-of-the-art solutions.