High-Performance Large Language Model Inference Framework for NVIDIA Edge Platforms
Overview | Support Matrix | Quick Start | Performance | Documentation | Roadmap
- [2026/09] TensorRT Edge-LLM completes the MLPerf Edge Agentic benchmark 6.4x faster on Jetson AGX Thor.
- [2026/09] Release 0.11.0 adds published Python wheels, Muse-Glimmer, experimental pi0.5, Hunyuan dense models, guided decoding, opt-in in-flight batching, native provider Jinja chat templates, and GPU-free CuTe DSL artifact generation.
- [2026/09] Release 0.10.1 adds experimental TP=2 inference on Dual NVIDIA DGX Spark and redesigns the experimental OpenAI-compatible server for faster cold launches and lower memory usage.
- [2026/08] TensorRT Edge-LLM 0.10.0 adds Day-0 support for Qwen3.8-27B.
- [2026/08] Release 0.10.0 adds support for NVIDIA Nemotron-3.5 Lightning with MTP and DFlash, Cosmos3-Edge, DiffusionGemma, Nemotron-3.5-ASR, and DSpark speculative decoding, alongside an experimental direct TensorRT engine builder without ONNX export, multi-turn KV-cache reuse, and video input for the experimental OpenAI-compatible server.
- [2026/07] Support for the full Gemma 4 family (E2B / E4B / 12B / 26B-A4B / 31B — multimodal text + image + audio, with MTP), Qwen3-Omni and Nemotron-3 NVFP4, and DFlash speculative decoding (with DDTree for Qwen3 / Qwen3.5) landed across releases 0.9.0 and 0.9.1.
TensorRT Edge-LLM is NVIDIA's C++ inference runtime for text, vision, audio, speech, and action models on NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark. The supported frontend exports Hugging Face checkpoints to ONNX for C++ engine building; an experimental direct frontend builds engines from checkpoints without ONNX. Both paths use the same C++ deployment runtimes.
For a supported target, install a published Python wheel
without compiling Edge-LLM. Use tensorrt-edgellm[runtime] for the high-level
Python API and HTTP serving. See Optional Python dependencies
for additional tools, or Install without optional dependencies
for the base installation.
Follow the Quick Start Guide to start a chat or send an inference request. Checkpoint IDs are listed in Supported Models.
Install from PyPI (requires a supported CUDA/TensorRT stack):
python -m pip install "tensorrt-edgellm[runtime]"The repository ships agent skills in skills/. Agent hosts
that support .agents/skills (including Codex and Cursor) discover them inside
a clone; Claude Code uses the .claude/skills link. Set up the project and let
the agent drive:
git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git && cd TensorRT-Edge-LLM
claude "Set up TensorRT Edge-LLM on this machine with the edge-llm-install skill. Ask before system-wide changes." # or: codex "..."To use the skills outside a clone, ask: Install the Edge-LLM agent skills for <Claude Code|Codex> with the edge-llm-install skill.
Each skill states the inputs it needs and the evidence it reports before it declares success.
| Task | Example prompt | Skill |
|---|---|---|
| Install, update, or remove | Install Edge-LLM from a wheel on this Jetson Thor and verify it. |
edge-llm-install |
| Serve a model | Serve <org/model> through the OpenAI-compatible API, send a streaming request, and stop it after the smoke test. |
edge-llm-serve |
| Convert and run a model | Export <org/model>, build the required engines, run C++ inference, and compare with the provider reference. |
edge-llm-convert-model |
| Add or customize a model | Add support for <org/model>; start from its config.json and checkpoint tensors. |
edge-llm-add-model |
| CuTe DSL kernel | Add a CuTe DSL kernel variant for sm_<NN> and validate it on freshly generated artifacts. |
edge-llm-cutedsl-kernel |
| TensorRT plugin | Add a TensorRT plugin for <op> to both frontends. |
edge-llm-plugin-extension |
| Upgrade or backport | Migrate my deployment from Edge-LLM <old> to <new>. |
edge-llm-version-migration |
| Accuracy mismatch | Output differs from the Hugging Face reference; find the first failing layer. |
edge-llm-few-layer-test, then edge-llm-accuracy-debugging |
| Run or add tests | Add an end-to-end case for <org/model> and run only that case. |
edge-llm-e2e-test, edge-llm-cpp-test |
Not sure which skill applies? Ask the agent to list the skills in skills/ and pick one.
- Overview - What is TensorRT Edge-LLM and key features
- Official Support Matrix - Platform, JetPack, DriveOS, CUDA, TensorRT, and TensorRT Edge-LLM compatibility
- Supported Models - Complete model compatibility matrix
- Checkpoint Exporter - Recommended ONNX export pipeline
- Experimental Direct Engine Builder - Build all model components directly from a checkpoint
- Installation - Set up quantization,
tensorrt_edgellm, and the C++ runtime - trtedge CLI - Chat, serve models, build engines, export ONNX, and prepare checkpoints
- Quick Start Guide - Run your first inference in ~15 minutes
- Examples - End-to-end workflows
- Quantization - Create quantized checkpoints for
tensorrt_edgellm - Python API and OpenAI-Compatible Server - vLLM-style API and OpenAI-compatible server
- Input Format Guide - Request format and specifications
- Chat Template Format - Chat template configuration
- Quantization Package Design - Quantization package architecture
- Engine Builder - Building TensorRT engines
- C++ Runtime Overview - Runtime system architecture
- Customization Guide - Customizing TensorRT Edge-LLM for your needs
- TensorRT Plugins - Custom plugin development
- Tests - Comprehensive test suite for contributors
See the Performance Benchmarks page for released benchmark results covering LLM and VLM prefill, generation throughput, memory usage, and EAGLE speculative decoding speedups.
🚗 Automotive
- In-vehicle AI assistants
- Voice-controlled interfaces
- Scene understanding
- Driver assistance systems
🤖 Robotics
- Natural language interaction
- Task planning and reasoning
- Visual question answering
- Human-robot collaboration
🏭 Industrial IoT
- Equipment monitoring with NLP
- Automated inspection
- Predictive maintenance
- Voice-controlled machinery
📱 Edge Devices
- On-device chatbots
- Offline language processing
- Privacy-preserving AI
- Low-latency inference
- TensorRT Edge-LLM Jetson AI Lab tutorial
- Run AI Models for Edge Application Development with Just a Few Commands
- Maximizing Memory Efficiency to Run Bigger Models on NVIDIA Jetson
- Build Next-Gen Physical AI with Edge-First LLMs for Autonomous Vehicles and Robotics
- Accelerate AI Inference for Edge and Robotics with NVIDIA Jetson T4000 and NVIDIA JetPack 7.1
- Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM
Follow our GitHub repository for the latest updates, releases, and announcements.
- Documentation: Full Documentation
- Quick Start: Quick Start Guide
- Roadmap: Developer Roadmap
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Forums: NVIDIA Developer Forums
We welcome contributions! Please see our Contributing Guidelines for details.
