Even though Triton supports inference across various platforms such as cloud, data center, edge and embedded devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia, it does so by relying on the backends. Note that not all Triton backends support every platform. The purpose of this document is to go over what all compute platforms are supported by each of these Triton backends. GPU in this document refers to Nvidia GPU. See GPU, Driver, and CUDA Support Matrix to learn more about supported GPUs.
The table below describes target device(s) supported for inference by each backend on different platforms.
| Backend | x86 | ARM-SBSA |
|---|---|---|
| TensorRT | ✔️ GPU ❌ CPU |
✔️ GPU ❌ CPU |
| ONNX Runtime | ✔️ GPU ✔️ CPU |
✔️ GPU ✔️ CPU |
| TensorFlow | ✔️ GPU ✔️ CPU |
✔️ GPU ✔️ CPU |
| PyTorch | ✔️ GPU ✔️ CPU |
✔️ GPU ✔️ CPU |
| OpenVINO | ❌ GPU ✔️ CPU |
❌ GPU ❌ CPU |
| Python1 | ✔️ GPU ✔️ CPU |
✔️ GPU ✔️ CPU |
| DALI | ✔️ GPU ✔️ CPU |
✔️ GPU2 ✔️ CPU2 |
| FIL | ✔️ GPU ✔️ CPU |
Unsupported |
| TensorRT-LLM | ✔️ GPU ❌ CPU |
✔️ GPU ❌ CPU |
| vLLM | ✔️ GPU ✔️ CPU |
Unsupported |
Only TensorRT and ONNX Runtime backends are supported on Windows.
| Backend | x86 | ARM-SBSA |
|---|---|---|
| TensorRT | ✔️ GPU ❌ CPU |
✔️ GPU ❌ CPU |
| ONNX Runtime | ✔️ GPU ✔️ CPU |
✔️ GPU ✔️ CPU |
Following backends are currently supported on Jetson Jetpack:
| Backend | Jetson |
|---|---|
| TensorRT | ✔️ GPU ❌ CPU |
| ONNX Runtime | ✔️ GPU ✔️ CPU |
| TensorFlow | ✔️ GPU ✔️ CPU |
| PyTorch | ✔️ GPU ✔️ CPU |
| Python1 | ❌ GPU ✔️ CPU |
Look at the Triton Inference Server Support for Jetson and JetPack.
Currently, inference on AWS Inferentia is only supported via python backend where the deployed python script invokes AWS Neuron SDK.
Footnotes
-
The supported devices for Python Backend are mentioned with respect to Triton. The python script running in Python Backend can be used to execute inference on any hardware if there are available python APIs to do so. AWS inferentia is one such example. Triton core is largely unaware of the fact that inference will run on Inferentia. ↩ ↩2
-
In case of ARM-SBSA, some operations are not fully supported. ↩ ↩2