Run AI Models for Edge Application Development with Just a Few Commands
Source
developer.nvidia.com
Author
Shara Tibken
Date
Why it matters
Gives a concrete local-serving path for edge models and a measured speculative decoding speedup (about 1.7x) without extra peak memory, useful for prototyping on-device agents.
Key takeaways · AI-distilled
NVIDIA TensorRT Edge-LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → is a lightweight C++ inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → runtime that uses the same framework for development on DGX Spark and deployment on Jetson Thor, DRIVE AGX Thor and IGX Thor.
The trtedge CLI downloads a model and builds TensorRT engines: trtedge chat opens a terminal session and trtedge serve exposes a local OpenAI-compatible API. The first Nemotron-3.5 Lightning launch downloads about 21GB; later launches reuse cached engines.
With DSpark speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → at three draft tokens, NVIDIA measured mean generation speed rising from 73.4 to 126.9 tokens/s on HumanEval prompts and from 73.3 to 108.5 on SpeedBench 1K, with median time to first token nearly unchanged.
NVIDIA's figures come from a narrow setup: DGX Spark 64GB, NVFP4 with thinking enabled, batch size one, one streamingSending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.Full definition → request and a 128-token output cap. Peak system memory with DSpark was 27.9GB including the OS.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
articleFrontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA JetsonElizabeth Goodman
articleCode, Test, and Deploy on One Machine with NVIDIA RTX SparkElizabeth Goodman
articleTensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX ThorElizabeth Goodman