TensorRT LLM optimizes inference for LLMs and Visual Gen models with specialized kernels for common operations, an efficient runtime, and a pythonic framework that enables you to customize and extend the system.
Architecture | Performance | Examples | Documentation | Roadmap
-
[10/14] KV Cache Compression in TensorRT LLM ✨ ➡️ link
-
[09/22] GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K ✨ ➡️ link
-
[09/02] Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM ✨ ➡️ link
-
[08/20] Evaluating Agentic Serving with Trace Replay and Job-Level Metrics ✨ ➡️ link
-
[07/17] DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM ✨ ➡️ link
-
[07/01] Scaling Video Generation Across NVL72 Rack with TensorRT-LLM ✨ ➡️ link
-
[05/15] Joint Optimization of Agent Applications and TensorRT-LLM ✨ ➡️ link
-
[04/03] Tuning CUDA Graph Batch Sizes for Higher Output Throughput ✨ ➡️ link
-
[04/03] DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72 ✨ ➡️ link
-
[03/16] Optimizing MoE Communication with One-Sided AlltoAll Over NVLink ✨ ➡️ link
-
[03/04] Sparse Attention in TensorRT LLM ✨ ➡️ link
-
[02/06] Accelerating Long-Context Inference with Skip Softmax Attention ✨ ➡️ link
-
[01/09] Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs ✨ ➡️ link
Previous Blogs
* [10/13] Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary) ✨ [➡️ link](https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog14_Scaling_Expert_Parallelism_in_TensorRT-LLM_part3.html)-
[09/26] Inference Time Compute Implementation in TensorRT LLM ✨ ➡️ link
-
[09/19] Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly ✨ ➡️ link
-
[08/29] ADP Balance Strategy ✨ ➡️ link
-
[08/05] Running a High-Performance GPT-OSS-120B Inference Server with TensorRT LLM ✨