Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard

Building an AI agent does not end with choosing a single model. Each model has its own strengths, weaknesses, and cost profile, which can shift from one workload to another—or even within the same workload. For example, an agentic task may need classification for one step, reasoning for the next, and a smaller model for routine follow-up tasks. Sending every request to the largest model can increase cost and latency, while sending every request to a smaller model can reduce quality on complex tasks.

Model routing addresses this challenge by orchestrating specialized and frontier models so that each task uses the model best suited to each task. NVIDIA NeMo Switchyard makes this complex engineering problem practical for agent workloads, so developers can route work across models without rebuilding their applications around each provider or model choice.

At runtime, a router evaluates each request and its available context, then sends the work to the model that best suits the task's requirements, constraints, and policies. Depending on the workload, this system of models may improve accuracy and reduce cost compared with using the most capable model for every request.

NeMo Switchyard provides a library for applying multiple routing approaches. This post explores how NeMo Switchyard enables developers to apply a system-of-models approach and build more efficient, controllable agents better suited for real AI workflows.

Video 1. Learn how NVIDIA NeMo Switchyard uses live signals to route each agent workflow step across a developer-configured pool of models

Consider a system of models performing a computer-use task measured by the Terminal-Bench Hard benchmark (Figure 1). While DeepSeek V4 has the highest overall accuracy in this example, it isn't the best model for every task group. For instance, Kimi K2.6 is better suited to the ML and RL task groups, while Qwen3.5 397B A17B is preferable for math and science. The remaining six task groups should use DeepSeek V4. This approach can also be applied at the individual-task level or across phases of solving a single task.

Figure 1. Model accuracies on different task groups on the Terminal-Bench Hard with the Terminus agent

Cost and completion time further complicate the decision, as shown in Figure 2. Each model has its own costs associated with running or accessing the model, as well as its own verbosity profile, relating not just to tokens but also tool calls.

Figure 2. Model accuracy vs total prompt tokens (left) and total output tokens (right) on Terminal-Bench Hard with the Terminus agent (STD bars are also shown)

Building a router requires considering signals from various sources. Each routing algorithm helps determine where to derive those signals.

Model capabilities: Which model(s) can solve the task correctly.

Model cost profile: The latency and cost associated with each model.

Infrastructure: System-level signals that enable reliable and seamless handoffs.

Look at the request itself. A router can use the classification to match requests based on topic or estimated difficulty. For example, a classifier can identify the topic of a query, match it to a model in the model pool, and route the query appropriately. An embedding model or feature crafter can be used to extract features from the query.

Look at the model states. A router can examine the model logprobs, cascades, assess agentic trace, model's residual stream, attention matrices, leverage, etc.

Look at the system. Pricing, latency, load, and agent-specific signals, including errors, are options. These signals can be used to evaluate model routers or as real-time routing signals.

Importantly, a router must consider not only which signals to use, but also when and where to evaluate them. For instance, for a multi-turn agent task, the router may route each full request to a specific model or route at each step. The whole system may share a pool, or sub-agents may use specialized model pools for the tasks.

The answers to these questions depend on several factors, including the use case, deployment complexity, error tolerance, latency, or throughput constraints. The system also requires infrastructure for a seamless and invisible handoff to the agent/user.

NeMo Switchyard solves these challenges with an intelligent orchestration layer that supports multiple routers. Developers can also bring their own routing algorithms or customization data to NeMo Switchyard.

Routing algorithms produce signals that inform routing decisions across models with different strengths. The system also needs infrastructure that can take a router's answer, send the request to the chosen model, and carry the response back to the application.

This starts with NeMo switchyard-libsy, the provider-agnostic SDK behind NeMo Switchyard. It represents requests, defines the models available to a system, and manages calls to the selected model. Each model target has a semantic name, while the client behind it maps that name to the provider endpoint and model ID. This separation keeps the routing logic independent of a specific provider.

Figure 3. Architecture diagram for NeMo Switchyard

NeMo Switchyard can carry routing state across an agent's session when a policy requires it. It can also retain information from earlier turns, such as tool results or an affinity decision, and make that context available for later routing decisions. A route can remain stateless when that history isn't needed.

This separation is important because model deployments change. A team may update a model, move it to another endpoint, or use a different provider without changing the routing integration. NeMo Switchyard can make a model call through the target client or return the call to the host application. This gives an agent runtime, inference platform, or gateway control over how requests are served while retaining the same routing contract.

The NeMo Switchyard server is a reference for making routing available through common APIs, to simulate an LLM gateway. It accepts OpenAI, Anthropic, and Responses API requests, translates them into the internal NeMo Switchyard request format, and returns the expected response format. It also records the selected model, decision rationale, token usage, latency, and call outcomes, so teams can inspect a running route.

With the infrastructure in place, the next step is looking at the routing approaches that use it to make decisions. NeMo Switchyard offers both tuning-free and tunable routers.

NeMo Switchyard includes several tuning-free routers that make decisions without training on workload-specific data, including the LLM classifier, stage router, and escalation router.

The LLM classifier uses an LLM as a judge to select a candidate LLM and maintains session affinity with that model across later turns. This avoids repeatedly reclassifying work that has not materially changed throughout the arc of an agent solving a task.

This approach fits headless and domain-specific systems. A team can route coding, mathematical, or healthcare work tasks to selected model targets, while NeMo Switchyard supplies the routing and state-management pieces.

A coding agent moves through different stages. Early on, it explores the codebase and recovers from errors. Later it settles into a more mechanical implementation. These stages require different levels of model capability, which the stage router uses to make routing decisions.

For each turn, the stage router examines recent tool activity to decide how much model capability the agent needs. Severe errors, repeated unproductive work, or prolonged exploration push the turn toward the capable model. Steady writes and edits, especially once tests are passed, favor the efficient model. If the signals are inconclusive, the router can consult an LLM judge before falling back to its configured default.

Escalation routing starts each conversation with a lower-cost model. An LLM judge monitors the progress of