TensorRT-LLM
https://github.com/nvidia/tensorrt-llm- Category
- AI Tools
- Rank
- No. 166Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- nvidia
- GitHub
- 14.6k stars
- Latest release
- v1.3.0rc25
- Date
About
NVIDIA's Python API for defining LLMs and running them on NVIDIA GPUs with state-of-the-art inference optimizations.
What it does
TensorRT-LLM turns language and visual generation models into optimized inference workloads. Specialized kernels and a modular PyTorch backend handle execution, while high-level interfaces support local, distributed, streaming, and server-based operation.
Why it's ranked here
This is a serious deployment system for teams committed to NVIDIA hardware. It combines optimized kernels, multi-node parallelism, quantized checkpoints, speculative decoding, visual generation, and multiple serving interfaces. That breadth comes with substantial operational complexity.
What's good
The public package surface loads components only when accessed, avoiding an eager import of most product modules. Users also get one framework spanning single-GPU inference, distributed execution, guided decoding, LoRA, prompt tuning, and visual generation.
Tradeoffs
The dependency surface is large and tightly coupled to CUDA, PyTorch, NCCL, and many pinned packages. Multi-GPU operation relies on MPI and documents several hang and recursive-spawn failure modes. Anonymous usage telemetry is enabled unless users choose an opt-out method.
How to use it well
Pick it for performance-focused inference on NVIDIA systems, especially when deployments may grow from one GPU to multiple nodes or need streaming servers. Budget time for dependency control, MPI troubleshooting, and model-specific tuning. It does not remove checkpoint licensing or authentication requirements.
Technical notes+
setup.py builds a platform-tagged binary distribution, selecting Windows DLLs or Linux shared objects and supporting extraction from precompiled wheels, directories, or remote archives. pyproject.toml configures setuptools plus Ruff, YAPF, isort, codespell, and autoflake. tensorrt_llm/__init__.py prepares the runtime environment before importing torch, maps public symbols through _LAZY_ATTRS, caches resolved attributes, and then calls _init(). tensorrt_llm/grpc/__init__.py conditionally exposes protobufs, request management, and a streaming servicer. tensorrt_llm/usage/__init__.py documents telemetry controls. requirements.txt shows a broad CUDA, inference, serving, media, and distributed-computing dependency set.
Observed
- License
- Apache License 2.0
- Primary language
- Python package with bundled native shared libraries and CUDA source
- Packaging
- Setuptools binary distribution with precompiled wheel support
- Interfaces
- High-level Python API, CLI serving, OpenAI-style servers, and gRPC
- Platform support
- NVIDIA CUDA systems, with Windows and non-Windows binary package layouts
- Deployment scope
- Single-GPU, multi-GPU, and multi-node inference
Read from README.md, setup.py, pyproject.toml, requirements.txt, docs/README.md, docs/source/llm-api/index.md, tensorrt_llm/__init__.py, triton_kernels/__init__.py, tensorrt_llm/grpc/__init__.py, tensorrt_llm/media/__init__.py, tensorrt_llm/serve/__init__.py, tensorrt_llm/tools/__init__.py, tensorrt_llm/usage/__init__.py, tensorrt_llm/_torch/__init__.py.
What it can do
Define and configure large language models
Model architecture specifications and parameters → Configured LLM model ready for deployment
Optimize LLM inference performance on NVIDIA GPUs
Pre-trained language model → GPU-optimized model with improved inference speed
Run text generation inference
Text prompts and configured LLM model → Generated text responses
Batch process multiple inference requests
Multiple text prompts → Batch of generated text outputs
Convert models to TensorRT optimized format
Standard LLM model files → TensorRT-optimized model engines
Manage GPU memory allocation for large models
Model size and available GPU memory specifications → Optimized memory configuration for efficient inference
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.