- Category
- AI Tools
- Rank
- No. 171Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- nvidia
- GitHub
- 13.3k stars
- Latest release
- v11.2
- Date
About
NVIDIA's SDK for high-performance deep learning inference on NVIDIA GPUs — optimizes trained models for low-latency production deployment.
What it does
TensorRT provides a set of components for bringing existing neural-network models into NVIDIA’s inference stack. It accepts several import routes, including ONNX, Torch-based integrations, Hugging Face tooling, and direct network definitions. The open repository supplies an ONNX parser, extensible operators, model-graph utilities, quantization tools, and usage samples.
Why it's ranked here
TensorRT is compelling when NVIDIA hardware is already the deployment target. It combines a packaged Python entry point with lower-level source components, documented import choices, model support guidance, and containerized builds. The catch is important: this repository exposes only a subset of the general release, so the source tree is not the whole product.
What's good
The project supports several practical paths from model ecosystems into one inference workflow. Strongly typed networks and explicit quantization make key behavior deliberate rather than implicit. Its surrounding tools cover graph editing, framework-specific quantization, validation workflows, samples, and multi-device builds. Prebuilt Python packaging lets users start without compiling the open components.
Tradeoffs
The stack is tied to CUDA and NVIDIA deployment environments. Building the open components requires the separately distributed general-release libraries plus a substantial native toolchain. Recent API cleanup removed weak typing, implicit quantization, older plugins, an analysis tool, and older Python bindings, creating migration work. Linux, Windows, and Arm builds follow different setup paths.
How to use it well
Choose TensorRT after selecting NVIDIA GPUs and an import route that supports your model family. Start with the Python package, then use the samples and support matrix to validate conversion. Build the open components when custom operators, parser work, platform ports, or deeper integration justify the toolchain. It does not present itself as a model-training framework or a hardware-neutral runtime.
Technical notes+
README.md defines the repository as an OSS subset containing TensorRT plugins, the ONNX parser, samples, extensions, and bug fixes, while requiring the TensorRT GA build for source compilation. python/packaging/tensorrt_build_backend/__init__.py implements a PEP 517 backend over setuptools, with dynamic binding, standalone binding, library, and frontend wheel names plus configurable Python and platform tags. scripts/convert_te_onnx_to_trt_onnx.py rewrites TensorRT FP8 quantize and dequantize custom operators into ONNX opset 19 nodes and can remove neighboring casts. tools/onnx-graphsurgeon/onnx_graphsurgeon/__init__.py exposes graph, node, tensor, import, export, and pattern APIs. tools/Polygraphy/polygraphy/__init__.py identifies a separate validation-oriented package within the tree.
Observed
- License
- Apache License 2.0
- Install surface
- Prebuilt Python package available through pip as tensorrt
- Interfaces
- Python package, native OSS components, command-line scripts, and CMake builds
- Languages shown
- Python, C, C++, CUDA, CMake, and shell sources are recognized by repository tooling
- Platform support
- Build instructions cover Linux and Windows on x86-64, Linux on Arm, Jetson, and DriveOS cross-compilation
- Repository scope
- Open-source plugins, ONNX parser, samples, extensions, and bug fixes form a subset of the general-release product
- Packaging structure
- Custom PEP 517 backend produces binding, standalone binding, libraries, and frontend wheel variants
Read from README.md, scripts/copyright-scan.py, scripts/convert_te_onnx_to_trt_onnx.py, tools/Polygraphy/polygraphy/__init__.py, python/packaging/tensorrt_build_backend/__init__.py, tools/onnx-graphsurgeon/onnx_graphsurgeon/__init__.py, tools/pytorch-quantization/pytorch_quantization/__init__.py, tools/polygraphy-extension-trtexec/polygraphy_trtexec/__init__.py, tools/tensorflow-quantization/tensorflow_quantization/__init__.py, tools/Polygraphy/polygraphy/mod/__init__.py, tools/Polygraphy/polygraphy/cuda/__init__.py, tools/Polygraphy/polygraphy/func/__init__.py, tools/Polygraphy/polygraphy/json/__init__.py, tools/Polygraphy/polygraphy/util/__init__.py.
What it can do
Optimize trained deep learning models for inference
Trained neural network models (ONNX, TensorFlow, PyTorch) → Optimized TensorRT engine for fast inference
Reduce model inference latency
Deep learning model and target hardware specifications → Low-latency optimized model
Convert models to TensorRT format
Models in various formats (ONNX, TensorFlow SavedModel, PyTorch) → TensorRT serialized engine file
Perform mixed precision optimization
Full precision neural network model → Mixed precision model (FP16/INT8) with maintained accuracy
Deploy optimized models for production inference
TensorRT optimized engine and inference requests → Real-time prediction results
Calibrate models for INT8 quantization
Neural network model and calibration dataset → INT8 quantized model with calibration tables
Intel on TensorRT
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.