TensorRT-LLM: Maximum Throughput for NVIDIA GPUs
TensorRT-LLM is an open-source library that optimizes Large Language Model inference for high performance on NVIDIA hardware.
When every millisecond counts, TensorRT-LLM is the answer. Developed by NVIDIA, it is a specialized library that optimizes the entire LLM execution pipeline for maximum throughput and minimum latency on H100, A100, and RTX GPUs.
State-of-the-Art Optimizations
TensorRT-LLM incorporates the latest inference techniques, including In-flight Batching, PagedAttention, and FP8 quantization. These optimizations allow for significantly higher request density, enabling organizations to serve more users with fewer GPUs.
Seamless Model Support
The library provides pre-optimized configurations for the most popular models, including Llama, Mistral, and Falcon. It integrates with standard deployment tools like Triton Inference Server, making it easy to incorporate world-class performance into your production AI stack.