TGI: Deploying LLMs with Hugging Face

Text Generation Inference (TGI) is the battle-tested toolkit used by Hugging Face to power their own Inference Endpoints. It is designed for high-performance, production-ready deployment of the most popular open-source models.

Optimized for Performance

TGI includes advanced features like tensor parallelism for multi-GPU serving, token streaming for real-time responses, and optimized CUDA kernels for popular architectures like Llama, Falcon, and StarCoder.

Production-Ready Serving

With built-in support for Prometheus metrics, health checks, and a production-grade web server, TGI provides everything you need to turn a model from the Hugging Face Hub into a scalable, reliable API for your applications.