From Build to Benchmark: ONNX Model Serving with Triton Inference Server on AMD GPUs

6 min read. | 1520 total words.

Triton Inference Server is an open-source platform designed to streamline AI inferencing. It supports the deployment, scaling, and inference of trained models from multiple frameworks, including ONNX Runtime, TensorFlow, PyTorch, and others. It runs across cloud, data center, and edge environments, making it adaptable for diverse AI workloads.

Framework flexibility: You can deploy models from different frameworks (see the Triton Inference Server Backend documentation) regardless of underlying infrastructure. This flexibility allows you to run multiple models or multiple instances of the same model on the same hardware, improving resource utilization.

Framework flexibility: You can deploy models from different frameworks (see the Triton Inference Server Backend documentation) regardless of underlying infrastructure. This flexibility allows you to run multiple models or multiple instances of the same model on the same hardware, improving resource utilization.

Hardware and deployment versatility: Triton is optimized for both GPU and CPU-based environments, allowing deployment on a variety of hardware. You can use Triton Inference Server in the cloud, in data centers, or on edge devices.

Hardware and deployment versatility: Triton is optimized for both GPU and CPU-based environments, allowing deployment on a variety of hardware. You can use Triton Inference Server in the cloud, in data centers, or on edge devices.

Performance optimization: Triton Inference Server enhances inference performance through dynamic batching, which aggregates multiple inference requests into batches to optimize processing and enable concurrent model execution. This capacity allows multiple models to run simultaneously, which is important for real-time applications that require minimal latency.

Performance optimization: Triton Inference Server enhances inference performance through dynamic batching, which aggregates multiple inference requests into batches to optimize processing and enable concurrent model execution. This capacity allows multiple models to run simultaneously, which is important for real-time applications that require minimal latency.

This blog is part of a series on deploying AI models with Triton Inference Server on AMD GPUs. In a previous blog, Triton Inference Server with vLLM on AMD GPUs, you can find a step-by-step guide for serving large language models using the vLLM backend. More recently, Serving CTR Recommendation Models with Triton Inference Server using the ONNX Runtime Backend introduced the ONNX Runtime and Python backends along with performance benchmarks comparing AMD Instinct MI355X and NVIDIA B200 GPUs.

In this blog, you will learn how to set up a Triton Inference Server with the ONNX Runtime backend on AMD GPUs, using ROCm and MIGraphX as the graph optimization accelerator. You will walk through the process of building the server from source, configuring the model repository, running inference, and benchmarking performance with the ResNet50-v2 image classification model from the ONNX Model Zoo.

AMD GPU: See the ROCm documentation page for supported hardware and operating systems. This blog was tested on a machine with AMD Instinct MI300X GPUs.

AMD GPU: See the ROCm documentation page for supported hardware and operating systems. This blog was tested on a machine with AMD Instinct MI300X GPUs.

ROCm 7.2+: See the ROCm installation guide for Linux for installation instructions.

ROCm 7.2+: See the ROCm installation guide for Linux for installation instructions.

Docker: See Install Docker Engine on Ubuntu for installation instructions.

Docker: See Install Docker Engine on Ubuntu for installation instructions.

ONNX models: This blog uses ResNet50-v2 from the ONNX Model Zoo. Instructions for downloading the model are provided in the Setting up the model repository section.

ONNX models: This blog uses ResNet50-v2 from the ONNX Model Zoo. Instructions for downloading the model are provided in the Setting up the model repository section.

A Triton Inference Server backend is the component responsible for executing an AI model during inference. Each backend wraps a specific machine learning framework such as ONNX Runtime, PyTorch, TensorFlow, or others. Each backend is implemented as a shared library and models are configured to use a specific backend through the model configuration file (config.pbtxt). For a list of all supported backends, see Where can I find all the backends that are available for Triton?. This blog focuses on the ONNX Runtime backend.

When you use ONNX Runtime as the backend on AMD GPUs, the inference pipeline relies on an execution provider (also called an execution accelerator) to bridge the gap between the framework and the hardware. An execution provider is a specialized software interface that translates a model’s computational graph into optimized instructions for a specific hardware architecture. On AMD Instinct GPUs, this role is filled by MIGraphX, AMD’s graph inference engine, which compiles and optimizes model graphs for high-performance execution. MIGraphX acts as a compiler that takes your generic ONNX model and produces a highly optimized set of instructions tailored for AMD hardware, similar to how a C compiler optimizes source code for a particular CPU architecture.

Triton Inference Server receives the inference request and routes it to the appropriate model.

Triton Inference Server receives the inference request and routes it to the appropriate model.

ONNX Runtime backend loads and manages the ONNX model.

ONNX Runtime backend loads and manages the ONNX model.

MIGraphX execution provider compiles and optimizes the model graph for AMD GPU hardware.

MIGraphX execution provider compiles and optimizes the model graph for AMD GPU hardware.

AMD GPU executes the optimized computation.

AMD GPU executes the optimized computation.

MIGraphX integration: When you enable the MIGraphX execution accelerator in the model configuration config.pbtxt, ONNX Runtime offloads graph compilation and execution to MIGraphX. This provides optimized kernel selection and memory management tailored for AMD Instinct GPUs.

MIGraphX integration: When you enable the MIGraphX execution accelerator in the model configuration config.pbtxt, ONNX Runtime offloads graph compilation and execution to MIGraphX. This provides optimized kernel selection and memory management tailored for AMD Instinct GPUs.

Model caching: MIGraphX compiles model graphs at load time, which can take several minutes for large models. You can cache the compiled models to disk using the ORT_MIGRAPHX_MODEL_CACHE_PATH environment variable. With model caching, subsequent server restarts will be much faster as no model compilation is needed.

Model caching: MIGraphX compiles model graphs at load time, which can take several minutes for large models. You can cache the compiled models to disk using the ORT_MIGRAPHX_MODEL_CACHE_PATH environment variable. With model caching, subsequent server restarts will be much faster as no model compilation is needed.

Dynamic batching: Triton supports dynamic batching, which groups incoming inference requests into batches of optimal sizes to improve GPU utilization and throughput. You can configure this through the dynamic_batching section in config.pbtxt.

Dynamic batching: Triton supports dynamic batching, which groups incoming inference requests into batches of optimal sizes to improve GPU utilization and throughput. You can configure this through the dynamic_batching section in config.pbtxt.

The config.pbtxt file is the model configuration file for Triton Inference Server, and you need one of these files for every model you deploy. It is written in Protocol Buffers text format and lives alongside the model file in the model repository. It defines the backend, input/output tensors, batching strategy, and optimization settings for each model. You will learn how to set up your model repository and write these configuration files in the sections below.

The ROCm-enabled Triton Inference Server and its components are maintained across several repositories. All components listed below are based on Triton Inference Server r25.12:

To deploy ONNX models on AMD GPUs, you need to build a Triton Inference Server Docker image with ROCm support and the ONNX Runtime backend. This section provides build instructions for Ubuntu 24.04 and Debian 12. The build process has two steps: building a base Docker image with the ROCm stack, and then building the Triton Inference Server image on top of it.

On Ubuntu 24.04#

Prerequisites (Ubuntu 24.04)#

ROCm 7.2 or compatible version installed on the host

ROCm 7.2 or compatible version installed on the host

Step 1: Build the base Docker image with Ubuntu 24.04 and ROCm 7.2.

Step 2: Build the Triton server Docker image.

ROCm 7.2 or compatible version installed on the host

ROCm 7.2 or compatible version installed on the host

Step 1: Build the base Docker image with Debian 12 and ROCm 7.2.

Step 2: Build the Triton server Docker image.

When you enable ROCm, the build automatically includes MIGraphX as the execution accelerator for the ONNX Runtime backend. The build compiles ONNX Runtime from the ROCm/onnxruntime fork and MIGraphX from ROCm/AMDMIGraphX.

A model repository is a directory structure that contains the models Triton Inference Server will serve. Each model is organized in a specific layout that Triton Inference Server scans and loads at startup. Similar to a library’s shelving system: each model has its own shelf (directory), with labeled versions and a catalog card (configuration file) describing its properties.

A version directory (1/) that holds the actual model file (model.onnx). Triton supports multiple versions, so you could have 1/, 2/, etc.

A version directory (1/) that holds the actual model file (model.onnx). Triton supports multiple versions, so you could have 1/, 2/, etc.

A configuration file (config.pbtxt) that defines the backend, input/output tensor names, shapes, data types, batching settings, and GPU optimization parameters.

A configuration file (config.pbtxt) that defines the backend, input/output tensor names, shapes, data types, batching settings, and GPU optimization parameters.

The ONNX Model Zoo repository structure may change over time. If the download URL above does not work, visit the ONNX Model Zoo directly and navigate to the ResNet50 model page.

Each model needs a config.pbtxt file that tells Triton Inference Server how to load and serve it. When using the ONNX Runtime backend on AMD GPUs, the configuration must specify onnxruntime as the backend and include the MIGraphX execution accelerator in the optimization block.

Create the configuration file for ResNet50-v2 at model_repository/resnet50_onnx/config.pbtxt:

With the model repository set up, you can start the Triton Inference Server container. Start the container with the docker run command below, which exposes the AMD GPU devices, sets MIGraphX optimization environment variables, and mounts the model repository.

Grant access to AMD GPU devices inside the container.

Grant access to AMD GPU devices inside the container.

Share host IPC namespace and network stack for GPU inter-process communication and direct port access.

Share host IPC namespace and network stack for GPU inter-process communication and direct port access.

Expose HTTP (8000), gRPC (8001), and metrics (8002) ports.

Expose HTTP (8000), gRPC (8001), and metrics (8002) ports.

Mount your model repository inside the container.

Mount your model repository inside the container.

Mount a directory for caching compiled MIGraphX models. This path must be outside the model repository (/models) so Triton does not try to load the cache as a model.

Mount a directory for caching compiled MIGraphX models. This path must be outside the model repository (/models) so Triton does not try to load the cache as a model.

Directory for caching compiled MIGraphX models. On subsequent restarts, the server loads from cache instead of recompiling.

Directory for caching compiled MIGraphX models. On subsequent restarts, the server loads from cache instead of recompiling.

Alternative cache path variable used by some ONNX Runtime versions. Set to the same value as ORT_MIGRAPHX_MODEL_CACHE_PATH.

Alternative cache path variable used by some ONNX Runtime versions. Set to the same value as ORT_MIGRAPHX_MODEL_CACHE_PATH.

Comma-separated list of batch sizes to pre-compile (e.g., "1,2,4,8"). Pre-compiling avoids recompilation when different batch sizes arrive at runtime.

Comma-separated list of batch sizes to pre-compile (e.g., "1,2,4,8"). Pre-compiling avoids recompilation when different batch sizes arrive at runtime.

Tells Triton where to find the model directories inside the container.

Tells Triton where to find the model directories inside the container.

Keeps the server running even if some models fail to load, allowing healthy models to serve requests.

Keeps the server running even if some models fail to load, allowing healthy models to serve requests.

The first time you start the server, MIGraphX will compile the model graphs for each batch size listed in ORT_MIGRAPHX_COMPILE_BATCHES. This can take several minutes depending on the model size and the number of batch sizes. Subsequent starts will load from the cache directory, making startup much faster.

With the Triton Inference Server running, you can now send inference requests to the ResNet50-v2 model. This section shows you how to set up a Python environment, install the Triton client library, and run a sample inference script.

The following Python script sends an inference request to the ResNet50-v2 model. For demonstration purposes, this example uses random input data. In a real application, you would preprocess an actual image (resize to 224x224, normalize with ImageNet mean and standard deviation, and convert to CHW (Channel-Height-Width tensor format)).

The class indices shown above correspond to ImageNet class labels (e.g., 504 = coffee mug, 968 = cup). Since the input image is random, the model output does not correspond to any meaningful classification result and is provided for illustration purposes only.

Triton Inference Server includes a tool called Performance Analyzer (perf_analyzer) that measures inference throughput and latency. It works by sending a sustained stream of requests to the server and collecting timing statistics over a measurement window.

perf_analyzer is available in the official Triton SDK container image (nvcr.io/nvidia/tritonserver:25.12-py3-sdk). Since it only sends requests over HTTP or gRPC, it does not require GPU access. It just needs network connectivity to the running Triton Inference Server.

Triton server URL.

Triton server URL.

Model name to benchmark.

Model name to benchmark.

Protocol to use (http or grpc).

Protocol to use (http or grpc).

Generate random input tensors matching the model’s expected shape.

Generate random input tensors matching the model’s expected shape.

Collect measurements over a 30-second window for stable results.

Collect measurements over a 30-second window for stable results.

Number of concurrent client requests in flight.

Number of concurrent client requests in flight.

Batch size per request.

Batch size per request.

After the measurement window completes, perf_analyzer prints a report similar in format to the example below. The exact numbers depend on the hardware, driver and ROCm versions, container build, model build, client/server placement, and overall system load. Treat the values shown only as a guide to interpreting the report layout, not as performance claims.

The throughput and latency values shown above are illustrative output from a single, untuned perf_analyzer run and are included solely to explain the structure of the report. They are not official benchmark results, are not intended for hardware or software comparisons, and should not be cited as performance claims. Your own results will vary with hardware, ROCm and driver versions, container and model build, configuration, and system conditions.

Client: Metrics as seen from the client side. Throughput is the number of inferences per second the server sustained. The latency percentiles (p50, p90, p95, p99) show how long individual requests took end-to-end, including network overhead.

Client: Metrics as seen from the client side. Throughput is the number of inferences per second the server sustained. The latency percentiles (p50, p90, p95, p99) show how long individual requests took end-to-end, including network overhead.

Server: Metrics reported by the Triton server itself. The Avg request latency breakdown shows where time is spent: queue wait, input preparation, actual model computation (compute infer), and output formatting. This breakdown is useful for identifying bottlenecks. For example, a large queue time suggests the server is saturated and could benefit from higher concurrency or more model instances.

Server: Metrics reported by the Triton server itself. The Avg request latency breakdown shows where time is spent: queue wait, input preparation, actual model computation (compute infer), and output formatting. This breakdown is useful for identifying bottlenecks. For example, a large queue time suggests the server is saturated and could benefit from higher concurrency or more model instances.

You can experiment with different batch sizes (-b 4, -b 8) and concurrency levels (--concurrency-range 1:4) to find the configuration that maximizes throughput for your workload.

In this blog, you walked through the deployment and serving of the ResNet50-v2 ONNX model using Triton Inference Server with the ONNX Runtime backend, powered by AMD Instinct MI300X GPUs and the ROCm software platform. You learned how to:

Build a Triton Inference Server Docker image with ROCm and MIGraphX support on both Ubuntu 24.04 and Debian 12.

Build a Triton Inference Server Docker image with ROCm and MIGraphX support on both Ubuntu 24.04 and Debian 12.

Set up a model repository with ONNX models and configure them with MIGraphX as the execution accelerator.

Set up a model repository with ONNX models and configure them with MIGraphX as the execution accelerator.

Start the Triton Inference Server with AMD GPU device access and MIGraphX optimization environment variables.

Start the Triton Inference Server with AMD GPU device access and MIGraphX optimization environment variables.

Send inference requests using the Python tritonclient library.

Send inference requests using the Python tritonclient library.

Use perf_analyzer to measure model throughput and latency across different batch sizes and concurrency levels.

Use perf_analyzer to measure model throughput and latency across different batch sizes and concurrency levels.

By using MIGraphX as the graph optimization engine within the ONNX Runtime backend, you can take full advantage of AMD Instinct GPU hardware for high-throughput, low-latency inference of ONNX models.

Triton Inference Server with vLLM on AMD GPUs: A step-by-step guide to serving large language models (Phi-2, Mistral, Llama-3) using the vLLM backend on AMD Instinct GPUs.

Triton Inference Server with vLLM on AMD GPUs: A step-by-step guide to serving large language models (Phi-2, Mistral, Llama-3) using the vLLM backend on AMD Instinct GPUs.

Serving CTR Recommendation Models with Triton Inference Server using the ONNX Runtime Backend: An overview of the ONNX Runtime and Python backend support, the version 25.12 upgrade, and performance benchmarks on AMD Instinct MI355X.

Serving CTR Recommendation Models with Triton Inference Server using the ONNX Runtime Backend: An overview of the ONNX Runtime and Python backend support, the version 25.12 upgrade, and performance benchmarks on AMD Instinct MI355X.

Thank you to the broader AMD team whose contributions made this work possible: Eliot Li, Ted Maeurer, David Rooney, Yao Liu, Christopher Austen, Khalique Ahmed, Titir Santra, Vish Vadlamani.

Third-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.