Scaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 2
Source
developer.nvidia.com
Author
Tanya Lenz
Date
Why it matters
Engineers serving models on NVIDIA GPUs can search TensorRT build configurations automatically, discarding routes that fail accuracy, and measure 1.14x to 1.63x gains over the default build. The post also covers caching and when to re-tune.
Key takeaways · AI-distilled
TensorRT 11.1 exposes more than 200 tunable compiler knobs. A build route is one combination of knob values, applied and inspected with IBuilderConfig::setBuildRoute, getBuildRoute and getAllBuildRoutes; a Python version is due in Torch-TensorRT 2.15.
Accuracy is a filter and speed is the ranking: routes that crash or miss a user-set accuracy tolerance against reference outputs are dropped, and the fastest remaining engine wins. The authors note the result is only validated for the supplied data, metric and threshold.
NVIDIA recommends fast search (one knob varied at a time) for broad scans, mixed search for larger expressions, and full Cartesian search only for a few chosen knobs. --dryRun lists routes before building, except in mixed mode, whose second phase depends on measurements.
A tuned route is specific to the model, shapes, GPU, CUDA or driver and TensorRT version, and may regress elsewhere. --tuningCacheFile with --continue resumes interrupted sweeps, and --saveAllEngines keeps every candidate engine for debugging at a storage cost.
articleAccelerating Mamba-2 with High-Performance Forward Pass Kernels in NVIDIA CUDA Tile
articleScale Agentic Coding Workflows with Meta’s Muse Spark Open on NVIDIA GB300 NVL72
articleOptimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym
articleNVIDIA Run:ai Is Now Open Source as Part of NVIDIA DSX AI Factory Platform
