Intel / article

TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor

Source: developer.nvidia.com
Author: Elizabeth Goodman
Date: Published Sep 16, 2026

Context

NVIDIA reports that its TensorRT Edge-LLM software completed the MLPerf Edge Agentic benchmark, which replays recorded software-engineering agent sessions, on a Jetson AGX Thor in 24 minutes 36 seconds, versus 2 hours 37 minutes for a llama.cpp reference run. Both ran the Qwen3.6-27B model from Qwen on the same hardware platform, with different formats. The 6.4-fold comparison covers 1,007 turns across 20 conversations.

One important mechanism is avoiding repeated processing of conversation history. The runtime restores cached state and processes the new suffix; roughly 96% of prompt tokens came from cache in this workload. Multi-token prediction separately proposes several continuations for the main model to verify together, reducing generation steps.

Key takeaways · AI-distilled

KV-cache and recurrent-state reuse across agent turns served about 96% of prompt tokens from hot cache in NVIDIA's edge benchmark run, so the runtime only had to prefill roughly 0.5M of the 13.6M total prompt tokens across the trajectory.
Tree-based multi-token prediction (8 draft steps, top-2 candidates, 16-node tree) beat linear multi-token prediction by about 40% on decoding speed for function-calling workloads, because tool names and JSON syntax are predictable enough to verify multiple branches in one pass.
NVFP4 4-bit quantization of weights, activations, and even the language-model head, paired with FP8 for the , let a single Jetson AGX Thor edge device run Qwen3.6-27B at 52.33 tokens/sec while hitting 87.94% accuracy on the Berkeley Leaderboard.
MLPerf's new Edge Agentic benchmark replays real agent trajectories (20 conversations, 1,007 turns, growing to ~23.5K tokens of ) instead of single prompts, to measure how systems handle the long, tool-calling conversations agents actually produce.

Terms in this piece · Glossary

quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Read the source developer.nvidia.com