TensorRT Edge LLM
https://github.com/nvidia/tensorrt-edge-llm- Category
- AI Tools
- Rank
- No. 769Tools index
Previous survey · No. 761 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- nvidia
- GitHub
- 541 stars
- Latest release
- v0.10.1
- Date
About
Lightweight C++ LLM and VLM inference engine optimized for edge devices and physical AI.
What it does
TensorRT Edge LLM turns supported Hugging Face checkpoints into TensorRT engines for NVIDIA hardware. Its supported path optionally quantizes models, exports ONNX components, then builds engines. An experimental path compiles directly from local checkpoints. Both feed the same Python-free C++ deployment runtimes for text, image, audio, speech, and action workflows.
Why it's ranked here
This is a compelling deployment stack when the target is NVIDIA edge hardware. It combines checkpoint conversion, several quantization formats, custom GPU kernels, speculative decoding, streaming, cache reuse, and multimodal runtimes. The repository also backs those claims with contributor tests and detailed design documentation. Its narrow hardware scope and experimental higher-level interfaces make it a specialist choice, not a general inference default.
What's good
The deployment runtime has no Python dependency, while Python handles checkpoint loading, quantization, and export. Teams can choose portable ONNX intermediates with explicit component control or try direct checkpoint compilation. Runtime capabilities include paged attention, FP8 cache storage, LoRA adapters, streaming, cache reuse, and several speculative decoding methods. The project also documents compatibility matrices, benchmarks, examples, customization, and custom TensorRT plugins.
Tradeoffs
Deployment targets are limited to NVIDIA Jetson, DRIVE, and DGX Spark, with compatibility tied to specific platform software stacks. Feature support varies by model and deployment, and cache reuse has a narrower boundary than ordinary inference. Direct checkpoint compilation, the high-level Python API, and the OpenAI-compatible server are explicitly experimental. The Python package also pins core dependencies to exact versions, which can complicate integration into an existing environment.
How to use it well
Use it when production inference must run locally on supported NVIDIA systems, especially for robotics, vehicles, industrial devices, or offline assistants. Start from the official compatibility matrix, select a supported checkpoint and precision, then prefer the ONNX workflow for established deployments and inspect released benchmarks for the target workload. Treat direct compilation and serving as evaluation paths. It covers model preparation and inference deployment, not model training or hardware-neutral serving.
Technical notes+
pyproject.toml defines a setuptools package requiring Python 3.10 or newer, exact core dependency pins, optional tools and server extras, and seven CLI entry points. tensorrt_edgellm/model.py implements AutoModel.from_pretrained() dispatch from checkpoint metadata, while tensorrt_edgellm/__init__.py registers model-family implementations and exposes ONNX export. tensorrt_edgellm/config.py parses FP16, FP8, NVFP4, INT4 AWQ, INT4 GPTQ, INT8 SmoothQuant, and mixed-precision metadata. kernelSrcs/build_cutedsl.py AOT-compiles architecture-filtered CuTe DSL kernels into static archives. GPU-facing C++ coverage is visible in unittests/loggerTests.cpp, unittests/logitBiasTests.cpp, unittests/loraManagerTest.cpp, and unittests/eagleAcceptTests.cpp.
Observed
- License
- Apache License 2.0
- Languages
- C++ deployment runtime with Python checkpoint, quantization, export, build, and server tooling
- Packaging
- Setuptools Python package requiring Python 3.10 or newer, with tools and server extras
- CLI surface
- Commands cover quantization, ONNX export, experimental direct engine building, LoRA processing, and vocabulary reduction
- Interfaces
- C++ runtime, Python library, command-line tools, experimental Python API, and experimental OpenAI-compatible server
- Platforms
- NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark
- Testing structure
- Repository includes C++ unit tests using GoogleTest, including CUDA runtime behavior
Read from README.md, pyproject.toml, requirements.txt, docs/README.md, docs/source/overview.md, experimental/__init__.py, tensorrt_edgellm/model.py, unittests/loggerTests.cpp, tensorrt_edgellm/config.py, kernelSrcs/build_cutedsl.py, tensorrt_edgellm/__init__.py, tensorrt_edgellm/_version.py, unittests/logitBiasTests.cpp, unittests/loraManagerTest.cpp, unittests/eagleAcceptTests.cpp.
What it can do
Run large language model inference
Text prompts and model parameters → Generated text responses
Execute vision-language model inference
Images and text queries → Image analysis and descriptions
Optimize model performance for edge devices
Pre-trained LLM/VLM models → Optimized models with reduced latency and memory usage
Process AI workloads on resource-constrained hardware
Model inference requests → AI predictions and responses within edge device constraints
Deploy models for physical AI applications
Trained models and deployment configurations → Real-time AI inference for robotics and IoT devices
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.