TGI: Deploying LLMs with Hugging Face
Text Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models in production.
Text Generation Inference (TGI) is the battle-tested toolkit used by Hugging Face to power their own Inference Endpoints. It is designed for high-performance, production-ready deployment of the most popular open-source models.
Optimized for Performance
TGI includes advanced features like tensor parallelism for multi-GPU serving, token streaming for real-time responses, and optimized CUDA kernels for popular architectures like Llama, Falcon, and StarCoder.
Production-Ready Serving
With built-in support for Prometheus metrics, health checks, and a production-grade web server, TGI provides everything you need to turn a model from the Hugging Face Hub into a scalable, reliable API for your applications.