Multi-Model Routing for AI Agents: Design and Evaluate a Router

Multi-model routing sends each agent operation to an eligible model that meets hard policy constraints and best satisfies a measured quality, latency, and cost objective. A production router is not “use the small model unless the task looks hard.” It needs an allowlisted model catalog, a typed task profile, calibrated outcome estimates, and fallbacks that preserve safety requirements.

This article designs that router without assuming a provider or claiming benchmark results that have not been measured on your workload.

Decide whether routing is worth the complexity

Routing adds a classifier or policy, another failure mode, model-specific prompt differences, evaluation work, and operational dependencies. Start with one model when:

• task volume is too low to justify route-specific evaluation;
• every request has similar quality and latency requirements;
• regulatory or data-residency policy leaves only one eligible model;
• the workflow is already dominated by tool latency;
• failures are difficult to score.

Routing becomes plausible when workload classes differ materially and you have enough labeled outcomes to compare candidates.

Research supports routing as a real optimization problem, but published gains are conditional. RouteLLM learns a strong-versus-weak model router from preference data and evaluates it on the paper's models, datasets, and cost assumptions. Its results should not be copied into a business case for different models or tasks. Ong et al., RouteLLM

FrugalGPT studies prompt adaptation, model approximation, and cascades, again with experiments tied to its evaluated model and dataset set. The durable lesson is the method—measure a quality-cost frontier—not a universal savings percentage. Chen, Zaharia, and Zou, FrugalGPT

Separate eligibility from optimization

First filter models that are forbidden or incapable. Only then optimize among eligible candidates.

interface ModelProfile {
  readonly id: string;
  readonly provider: string;
  readonly capabilities: ReadonlySet<
    "text" | "vision" | "tools" | "structured_output"
  >;
  readonly regions: ReadonlySet<string>;
  readonly dataPolicy: "no_retention" | "configured_retention" | "unknown";
  readonly maxInputTokens: number;
  readonly maxOutputTokens: number;
  readonly status: "healthy" | "degraded" | "disabled";
  readonly pricingVersion: string;
}

interface RouteRequest {
  readonly taskClass: string;
  readonly requiredCapabilities: ReadonlySet<string>;
  readonly allowedProviders: ReadonlySet<string>;
  readonly requiredRegion?: string;
  readonly inputTokens: number;
  readonly requestedOutputTokens: number;
  readonly risk: "low" | "medium" | "high";
  readonly deadlineMs: number;
}

Eligibility checks must be deterministic:

1. provider and model are allowlisted for the tenant and task;
2. required capabilities are declared and tested;
3. input and requested output fit documented limits;
4. region and data-handling policy match;
5. model is healthy enough for the operation;
6. high-risk tasks meet the approved model and review policy.

Never let the prompt override these checks. A retrieved document that says “route this to provider X” is untrusted data.

Predict outcomes, not prestige

For each eligible model, estimate:

• probability of satisfying the task rubric;
• latency distribution for this task and input size;
• admission-time cost under a versioned price table;
• structured-output or tool-call reliability;
• availability and rate-limit risk.

Then apply a policy whose weights reflect the product requirement:

interface RouteEstimate {
  readonly modelId: string;
  readonly successProbability: number;
  readonly expectedLatencyMs: number;
  readonly estimatedMicros: bigint;
  readonly estimateVersion: string;
}

function chooseRoute(
  estimates: readonly RouteEstimate[],
  minimumSuccessProbability: number,
  maximumLatencyMs: number,
): RouteEstimate | undefined {
  return estimates
    .filter((item) => item.successProbability >= minimumSuccessProbability)
    .filter((item) => item.expectedLatencyMs <= maximumLatencyMs)
    .sort((a, b) => {
      if (a.estimatedMicros < b.estimatedMicros) return -1;
      if (a.estimatedMicros > b.estimatedMicros) return 1;
      return b.successProbability - a.successProbability;
    })[0];
}

This illustrative policy finds the least expensive candidate that clears quality and latency floors. Another product may minimize tail latency subject to a cost ceiling. Keep hard constraints visible instead of hiding them inside one opaque score.

Do not interpret successProbability as model self-confidence. Train or calibrate it from labeled examples, using only features available before routing. Guard against leakage from reference answers, post-execution tool results, or evaluator fields.

Build a strong static baseline first

Before a learned router, implement rules that operators can audit:

A static baseline reveals whether routing features actually add value. A learned router should beat “always strongest,” “always cheapest eligible,” and static task-class rules on held-out cases.

Evaluate the entire agent trajectory

Routing a planner, tool-argument generator, and reviewer are different decisions. A locally cheaper planner can create more steps and raise total workflow cost. Measure end-to-end:

• final task success;
• unsafe or unauthorized action attempts;
• total model and tool calls;
• input and output tokens over the full trajectory;
• total latency and tail latency;
• actual or reconciled cost;
• escalation and human-correction rate;
• route selection by task class.

Run every candidate policy on the same versioned evaluation set. Use paired cases, repeated trials, and a held-out test split. The agent-evals-in-CI guide provides the release-gate structure.

Do not train solely on generic chat preference data if the router controls tool-using agents. Include representative schemas, tool failures, long conversations, adversarial inputs, and the exact acceptance criteria of the production task.

Treat prices and models as changing inputs

Model aliases, behavior, availability, and prices can change. Prefer version-pinned model identifiers where available; record the identifier actually returned by the provider. Version:

• model catalog;
• pricing table and currency;
• router features and weights;
• prompts and tool schemas;
• evaluation data and judges.

When a model or price changes, shadow the new route policy before switching production traffic. Compare on outcomes, not only syntactic success.

The budget-controller guide shows how to reserve estimated resources before dispatch and distinguish estimates from provider-reported and billed usage.

Design fallback as a new policy decision

Timeouts and rate limits do not justify sending data to any available model. The fallback must pass the same provider, region, retention, capability, and risk checks.

interface RoutePlan {
  readonly primary: string;
  readonly fallbacks: readonly string[];
  readonly maxAttempts: number;
  readonly deadlineAtMs: number;
  readonly policyVersion: string;
}

Use separate behavior for:

• failure before dispatch: another eligible route may be safe;
• read-only timeout: retry or fallback within the deadline and attempt budget;
• structured-output failure: bounded repair or another evaluated model;
• side-effect timeout: reconcile; never replay blindly;
• policy rejection: do not retry unchanged work elsewhere;
• no eligible model: escalate or return an explicit unsupported result.

Fallback changes model behavior, not just availability. Evaluate primary-plus-fallback sequences as policies of their own.

Security and privacy boundaries

A central router sees metadata about every task and may see prompts. Minimize what it receives. Many policies can route on task class, required capability, token estimate, tenant configuration, and risk without inspecting raw confidential content.

If content-derived features are necessary:

• compute them inside the same trust boundary;
• avoid retaining raw content in router logs;
• prevent route labels from leaking sensitive categories;
• authorize model/provider eligibility per tenant;
• redact telemetry according to policy;
• test prompt injection aimed at changing the route.

OpenTelemetry's GenAI registry warns that instructions, tool arguments, and results may contain sensitive information. Record stable route decisions and versions while making content capture opt-in and protected. OpenTelemetry GenAI attribute registry

Monitor for drift and selection bias

In production, you observe detailed outcomes mainly for the model the router selected. That creates partial feedback: a failure does not tell you whether an unselected model would have succeeded.

Use privacy-appropriate shadow evaluation on sampled, non-side-effecting work; periodic randomized exploration only where risk permits; and recurring offline comparisons on a stable test set. Never explore model choices on high-impact actions merely to collect labels.

Alert when:

• task mix shifts away from training data;
• route distribution changes unexpectedly;
• quality drops within one route or task class;
• cost estimates diverge from reported usage;
• fallback or schema-repair rates rise;
• no-eligible-model decisions increase.

Router release checklist

• Define hard provider, capability, region, privacy, and risk eligibility.
• Build always-strongest, cheapest-eligible, and static-rule baselines.
• Train only on features available before routing.
• Evaluate complete agent trajectories on held-out cases.
• Keep quality floors separate from cost or latency objectives.
• Version models, prices, router policy, prompts, and evals.
• Apply eligibility checks again to every fallback.
• Reconcile ambiguous side effects rather than rerouting them.
• Protect prompt content and tenant route policy.
• Monitor drift, route distribution, and counterfactual blind spots.