How to Benchmark a Multi-Model Router

A trustworthy multi-model-router benchmark runs every eligible model on the same hidden evaluation cases, compares the router with fixed and oracle baselines, and reports paired quality, cost, latency, and constraint violations. It must include router overhead, fallbacks, abstentions, and failures. This article specifies the protocol; it does not present invented benchmark results.

State the routing decision precisely

A router can choose a model, provider, configuration, tool-enabled workflow, or cascade. Define:

• the exact candidate set and eligibility constraints;
• information visible at routing time;
• the objective or constraints;
• whether the router may abstain or fall back;
• whether it learns from outcomes;
• the unit of decision: request, turn, or complete task.

Do not call a cascade and a one-shot selector equivalent. A cascade may call a cheaper model, grade its answer, and escalate; its cost and latency include both attempts and the grader. A selector pays routing overhead once, then chooses.

The original RouteLLM paper studies routing between stronger and weaker models using preference data. FrugalGPT studies cascades over model APIs. These are useful design references, but their reported results do not predict your workload.

Freeze a benchmark manifest

Before running anything, publish a machine-readable manifest:

benchmark_id: support-routing-2026-07-18
decision_unit: complete_task
dataset_snapshot: sha256:...
prompt_version: git:...
candidates:
  - provider: example
    model: exact-model-id
    api_version: exact-version
    decoding: {temperature: 0}
router_version: git:...
tool_snapshot: container-digest
rate_card_date: 2026-07-18
trials_per_case: 3

Include system prompts, tool schemas, maximum output, safety settings, region, timeouts, retry rules, and provider concurrency. If an API silently aliases models, save the response model identifier when supplied. Changes after preregistration create a new run.

Build splits that resist leakage

Use development, calibration, and final test splits. Group related cases by source family, template, repository, customer, or time period before splitting so paraphrases cannot cross boundaries. Deduplicate near matches. Remove cases that appeared in the router’s training data when known, and disclose what cannot be verified.

The final test labels and outcomes must be hidden from the router. A router that sees a difficulty tag derived from the correct answer is leaking target information. Features must be available before the model call in production.

Cover both normal traffic and important slices: long context, tool use, coding language, locale, ambiguous requests, safety-sensitive tasks, out-of-domain inputs, and provider failure. Weight cases to a declared production distribution, but also publish unweighted slice results so rare critical failures remain visible.

NIST’s AI 800-3 report on evaluating generative AI distinguishes performance on a fixed benchmark from broader generalization and emphasizes statistical models and uncertainty. Name the population your dataset can support; do not label one suite “overall intelligence.”

Run the offline full matrix

For each test case, run every eligible candidate under identical prompt and tool conditions. Use multiple trials when outputs are stochastic. Randomize or interleave execution order to reduce time-of-day and provider-load bias.

Save for every candidate-case-trial tuple:

• raw outcome or artifact;
• deterministic correctness checks;
• blinded evaluator result and rubric version;
• provider status and finish reason;
• input, output, cached, and other billed usage categories;
• first-token and completion latency;
• tool calls, retries, and safety events;
• exact model and configuration identifiers.

The full matrix permits counterfactual evaluation: for a recorded router choice, retrieve that candidate’s observed outcome on the same case. It also supports an oracle ceiling. The oracle retrospectively selects the best candidate after seeing outcomes; label it clearly as non-deployable.

Do not use offline sequential latency as the only latency comparison. Provider load changes over time. Interleave calls within safe rate limits, repeat across windows, and separately run a live-routing test that includes contention and router service latency.

Calibrate the quality measurement

Use deterministic checks first: tests, schema validation, exact facts, tool-state verification, or policy rules. For subjective dimensions, blind evaluators to model and route, define an anchored rubric, randomize answer order, and measure agreement against expert labels.

If an LLM judges outputs, calibrate it on the target domain and retain human adjudication for disagreements and critical slices. Do not let the routed model grade itself. Re-run a fixed calibration set when the evaluator prompt or model changes.

Quality can be multi-dimensional. Report task success, factual support, instruction adherence, safety, and format validity separately before constructing a composite. Publish the composite weights; hidden weights are an undocumented business policy.

Price observed work, not hypothetical requests

Calculate variable cost from observed provider usage receipts and the rate card dated in the manifest. Include:

• router inference or feature computation;
• the selected model call;
• graders used inside a cascade;
• failed attempts and retries;
• fallback models;
• billed cache reads/writes;
• tool API and execution charges.

Keep raw usage alongside calculated currency so results can be repriced. Do not apply today’s price to an old run without labeling the analysis as a repricing. Report missing or ambiguous receipts.

Compare against baselines that expose value

At minimum include:

1. Always choose the strongest eligible candidate.
2. Always choose the least expensive eligible candidate.
3. A random or simple predeclared heuristic route.
4. The candidate router.
5. The retrospective oracle ceiling.

If the router was trained, include a held-out comparison and ablations for major feature groups. If it abstains, compare coverage at matched quality and cost. If it falls back, report first-choice and final outcomes separately.

The recent LLMRouterBench paper is an additional reference for unified measurement across quality, cost, and latency. Use its methodology as input, not as a substitute for disclosing your own protocol.

Report a frontier, not one magical score

Plot achievable quality against cost and latency. A route is Pareto-dominated if another is at least as good on all reported objectives and better on one. Also report policy and capacity constraints that cannot be traded away.

Core outputs should include:

• task success with uncertainty intervals;
• total and per-success cost distribution;
• first-token and completion latency distributions;
• abstention, fallback, timeout, and error rates;
• safety and policy-violation counts;
• route share and capacity by candidate;
• results by critical slice;
• quality-cost and quality-latency frontiers.

Use paired statistical analysis because candidates run on the same cases. Bootstrap by the independent sampling unit—often source family, not individual paraphrase—to avoid narrow intervals from correlated examples. Publish trial counts and the interval method. Do not declare small differences meaningful solely because a point estimate is larger.

Stress the router’s failure behavior

A production router needs tests that a static benchmark omits:

• one provider returns rate limits or times out;
• the preferred model is removed from eligibility;
• router features are missing or malformed;
• input distribution shifts beyond calibration;
• all candidates violate a latency or cost constraint;
• the routing service itself is unavailable;
• fallbacks threaten to exceed the total deadline or budget.

Define safe behavior before testing: default candidate, abstention, degraded mode, or fail closed. Ensure the model cannot edit eligibility rules or its own budget. Run the live router under representative concurrency so queueing, capacity limits, and fallback amplification appear.

Publish a benchmark card

A useful benchmark card includes the manifest, dataset construction, exclusions, slice distribution, label process, evaluator calibration, candidate eligibility, run dates, failures, raw usage, statistical method, and limitations. Release case-level results when privacy and licensing allow; otherwise release aggregate tables plus enough synthetic examples to reproduce the pipeline.

The strongest claim supported by this protocol is narrow: this router performed as reported on this frozen workload under these conditions. That disciplined claim is far more useful than an unsupported “best router” headline.