Run LLMs on edge devices with NVIDIA TensorRT Edge-LLM

NVIDIA has released a technical guide detailing how to run large language models on edge hardware using the new DGX Spark 64GB system. The tutorial, published on October 30, 2026, demonstrates how developers can prototype and serve AI agents locally with minimal setup. It highlights the integration of TensorRT Edge-LLM and speculative decoding techniques to maximize performance within tight memory constraints.

What happened

The core of this update is the availability of NVIDIA TensorRT Edge-LLM, a lightweight C++ inference runtime designed specifically for embedded workloads. This tool bridges the gap between development on the DGX Spark and deployment on production edge platforms like Jetson Thor, DRIVE AGX Thor, and IGX Thor. By providing a unified framework, NVIDIA aims to simplify the process of moving AI models from prototype to physical devices that monitor driver attention, coordinate robotics, or manage industrial fleets.

The tutorial focuses on running NVIDIA Nemotron-3.5 Lightning, a text-only model, on the DGX Spark 64GB. Developers can use the trtedge command-line interface to download models and build optimized TensorRT engines automatically. This process removes much of the manual configuration typically required for edge inference. The guide also shows how to serve the model via a local OpenAI-compatible API, allowing teams to test applications without needing cloud connectivity.

A significant portion of the release details the performance gains from DSpark speculative decoding. This technique uses a draft model to propose multiple tokens at once, which the main model then verifies in a single pass. On the DGX Spark 64GB, enabling DSpark with three draft tokens increased mean generation speed from 73.4 to 126.9 tokens per second on HumanEval prompts. Crucially, peak memory usage remained stable at 27.9GB, leaving room for the operating system and other processes.

How it works

TensorRT Edge-LLM operates by converting high-level model definitions into highly optimized inference engines tailored for specific hardware. The trtedge CLI handles the heavy lifting: it downloads the model weights, such as the 21GB Nemotron-3.5 Lightning package, and compiles them into TensorRT engines. These engines are cached locally, so subsequent runs skip the compilation step. The runtime supports Python environments, allowing developers to install the package via pip while maintaining access to system-level CUDA and TensorRT libraries.

DSpark speculative decoding accelerates generation by reducing the number of sequential steps required to produce text. Instead of generating one token at a time, the draft model predicts a sequence of future tokens. The main model then evaluates these predictions in parallel, accepting correct ones and correcting errors. This method significantly boosts throughput without increasing memory footprint, making it ideal for edge devices where resources are limited but latency matters.

Key details

• Hardware: The tutorial uses the NVIDIA DGX Spark 64GB configuration.
• Software Stack: Requires NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer, and Python 3.10–3.12.
• Model: NVIDIA Nemotron-3.5 Lightning (30B parameters, NVFP4 quantization).
• Performance: DSpark increases mean generation speed to 126.9 tokens/s on HumanEval prompts.
• Memory: Peak system memory usage stays at 27.9GB during inference with DSpark enabled.
• API: The server provides an OpenAI-compatible endpoint for local application testing.

Why it matters

For engineers building physical AI applications, the ability to prototype on hardware that mirrors production targets is critical. Edge devices often have strict power and memory budgets that data center GPUs do not face. By offering a development platform that shares the same inference runtime as deployment targets like Jetson Thor, NVIDIA reduces the risk of performance surprises when moving code to the field. This consistency helps teams iterate faster on agents that must perceive and act in real-time environments.

The performance metrics also highlight the maturity of edge inference tools. Achieving over 120 tokens per second on a 30-billion-parameter model while keeping memory under 28GB demonstrates that complex reasoning tasks are becoming viable on compact hardware. This opens up new possibilities for autonomous systems that need to process natural language instructions or generate reports locally without relying on intermittent cloud connections.

What you can do

• Install TensorRT Edge-LLM 0.11.1 in a virtual environment on your DGX Spark system.
• Verify your TensorRT version is 10.16.1.11 or newer before starting the engine build.
• Run the trtedge chat command to test Nemotron-3.5 Lightning with simple prompts.
• Start a local API server using trtedge serve to integrate the model into your applications.
• Enable DSpark speculative decoding with three draft tokens to boost generation speed.
• Explore other supported models, including vision-language and speech recognition options.