Efficient LLM Orchestration Framework

This paper addresses the practical challenges of self-hosting large language models (LLMs), which is becoming increasingly important for organizations. The proposed framework, Pick and Spin, offers a scalable and economical solution by integrating Kubernetes, adaptive scaling, and a hybrid routing module. The evaluation across multiple models, datasets, and inference strategies demonstrates significant improvements in success rates, latency, and cost compared to static deployments. This is a valuable contribution to the field, providing a practical approach to LLM deployment and management.

• •Pick and Spin is a practical framework for self-hosted LLM orchestration.
• •It uses Kubernetes, adaptive scaling, and hybrid routing.
• •Demonstrates improved success rates, lower latency, and reduced GPU cost.
• •Evaluated on multiple LLMs and datasets.