Choose an Open Model with an Evaluation Suite

Choose an open model by building a small, versioned evaluation suite around your product decisions—not by copying one leaderboard rank. Gate candidates on license, runtime compatibility, memory, context, and security first; then score the exact served artifacts on public capability tasks, private product cases, tool/structured behavior, safety, and a production-shaped load test. Select from the models that meet every hard requirement using a Pareto comparison of quality, latency, capacity, and operating cost.

This is a selection protocol, not a model ranking. HELM's core evaluation principles are broad coverage, multi-metric measurement, standardized comparison, and transparent reporting of what is missing (HELM, original HELM paper). Those principles matter more than any fixed task list because model families, chat formats, and public benchmarks keep changing.

Write the decision before naming candidates

Create a one-page evaluation charter:

product_routes:
  - support_answer_with_citations
  - read_only_tool_workflow
languages_and_domains: [...]
deployment:
  hardware_topology: ...
  runtime: ...
  maximum_prompt_and_output: ...
hard_gates:
  license_and_policy: ...
  minimum_product_quality: ...
  prohibited_effect_rate: 0
  schema_and_tool_requirements: ...
  latency_error_and_memory_slos: ...
tradeoffs_after_gates:
  quality_dimensions: ...
  capacity_and_cost_dimensions: ...
decision_owner: ...

The zero shown for prohibited effects is an example of a security invariant: a benchmark must never execute real effects. Define your own gates before results arrive. Do not create a weighted score after seeing which model it favors.

State what is out of scope. A model selected for English retrieval-assisted support is not thereby approved for medical advice, autonomous code deployment, or every supported language.

Gate artifacts before spending evaluation compute

For every candidate record:

• model repository, immutable revision, tensor hashes, and publisher;
• license text/version, acceptable-use conditions, and redistribution/deployment review;
• base versus instruct variant and training provenance disclosed by the model card;
• tokenizer revision, vocabulary, special tokens, and chat template;
• native and configured context behavior;
• runtime architecture support and required remote/custom code;
• intended precision, quantization artifact lineage, and kernel support;
• tool, reasoning, structured-output, and multimodal parser requirements; and
• weight, KV-cache, runtime, and host-resource feasibility.

Reject or quarantine unknown lineage. Do not set trust_remote_code merely to make a candidate load; inspect, pin, and sandbox repository code if the feature is genuinely required. Treat model and tokenizer caches as trusted executable input.

Use the GPU capacity-planning method to eliminate artifacts that cannot support the required live context/concurrency on target hardware. A theoretical bit-width estimate is not enough.

Build a layered suite

No public suite reflects one product. Use five layers with separate results.

1. General capability anchors

Select a small set of public, well-documented tasks representing reasoning, knowledge, language, code, or reading skills actually needed. Run them through a pinned framework such as EleutherAI's lm-evaluation-harness, which supports versioned task configurations and local/OpenAI-compatible inference endpoints (lm-evaluation-harness, CLI reference).

Do not import dozens of tasks and average them. For each task document:

why it predicts product value
dataset/task revision and license
few-shot count and exact prompt/template
generation and stop settings
metric implementation
known contamination or scoring limitations

Public benchmarks are useful anchors but may appear in training data. Treat high scores as evidence, not proof of generalization.

2. Private product cases

Build held-out cases from real failure modes and representative work, with sensitive data removed or governed. Include:

• common, long-tail, and adversarial inputs;
• explicit unknown/abstain cases;
• multilingual/domain strata;
• long and short context;
• conflicting or stale evidence;
• exact output and citation requirements; and
• regressions from incidents and user feedback.

Define a deterministic scorer where possible. For subjective properties, use a blinded human rubric with examples and adjudication. If a model judge is used, calibrate it against human labels, randomize candidate order, hide model identity, and report disagreement. Do not let the candidate grade itself.

Freeze a final test split before prompt tuning. Maintain a development split for templates and schemas; record every access to the final set.

3. Agent and interface behavior

Evaluate the model inside the exact orchestration it will use:

• correct tool selection and no-tool decisions;
• exact argument semantics and schema validity;
• clarification when required values are missing;
• parallel versus sequential call behavior;
• tool error, timeout, and malformed-result recovery;
• prompt injection inside retrieved/tool content;
• multi-turn state and stop behavior; and
• structured output parse, schema, and semantic correctness.

Berkeley's Function-Calling Leaderboard provides public executable function-call evaluation and includes single-turn, multi-turn, hallucination, and agentic categories in its current iteration (BFCL). Pin code/data revisions and supplement it with your exact tool schemas. Never connect evaluation tools to production credentials or networks; assert simulated effects.

4. Safety and policy

Derive hard tests from the threat model and product policy:

• no cross-tenant disclosure;
• no secret reproduction from canary context;
• no unauthorized write or high-impact action;
• correct approval boundary;
• resistance to indirect prompt injection;
• appropriate refusal without over-refusing benign cases; and
• bounded steps, tokens, time, and tool calls.

Measure the enforcement system, not only refusal text. A model may eagerly propose a prohibited action while a correct gateway blocks it; the model behavior and system invariant are separate metrics. Use synthetic tools and data.

NIST's Generative AI Profile is voluntary risk-management guidance, not a benchmark, but it supports documenting context, measurement limits, human oversight, and ongoing monitoring (NIST AI 600-1).

5. Serving and operational fit

Run the exact artifact, quantization, template, parser, and runtime on target hardware. Replay a fixed arrival trace across load points and record:

• successful requests and tokens per second;
• TTFT, TPOT, ITL, queue, and end-to-end percentiles;
• idle, warm, steady, and peak GPU/host memory;
• KV capacity, preemptions, OOMs, and errors;
• startup/readiness time and failure recovery;
• quality scores for outputs produced during the serving run; and
• energy or infrastructure cost only when measured with a documented method.

Use vLLM's benchmark client or an equivalent pinned load generator; its current CLI records detailed request and token latency metrics (vLLM Benchmark CLI). Documentation example numbers are not candidate results.

Normalize the inference contract

Model comparison is easily distorted by prompts. Decide whether the product will use:

• each publisher's recommended chat template; or
• one common prompt adapted to valid model-specific templates.

Publisher-recommended templates measure best supported deployment, but the prompt text can differ. A common raw string improves control but may be invalid for some instruct models. The defensible approach is to record rendered prompts/token IDs per model and state which question you are answering.

Pin:

• system/user/tool-role rendering;
• few-shot examples and order;
• tool and JSON schemas;
• temperature, top-p/top-k, penalties, seed behavior, and maximum tokens;
• EOS/stop handling;
• reasoning visibility/budget controls; and
• truncation side and context packing.

Do not award one candidate extra prompt tuning. Either use a predeclared generic recipe or allow the same bounded development budget for each and preserve the final templates.

Design the statistics before the run

For each metric define unit of analysis, denominator, missing/error handling, and interval method. Use case-level paired comparisons because all candidates see the same examples. Suitable methods include paired bootstrap intervals for continuous/task scores and exact paired tests for binary outcomes, with multiplicity awareness when making many claims.

Report:

• case counts and exclusions by stratum;
• point estimates and uncertainty intervals;
• paired win/loss/tie or delta distributions;
• errors and timeouts as outcomes, not deleted rows;
• sensitivity to prompt and sampling settings; and
• practical gate margins, not only statistical significance.

For sampled generation, predeclare repetitions and seeds where available. Temperature zero does not guarantee cross-stack bitwise determinism. Repeat infrastructure runs from clean processes and report run-to-run variance.

Preserve reproducibility

Write one immutable run bundle:

evaluation charter and gate versions
model/tokenizer/template/artifact hashes
runtime/container/driver/hardware manifest
task and dataset revisions
fully rendered prompts or protected hashes
resolved generation/server configuration
raw outputs, finish reasons, tool traces, and timings
scorer code/version and per-case results
aggregate notebook/report generated from raw results
known limitations and failed cases

Use cache keys that include every result-affecting field. Clear or isolate model conversations, prefix caches, and mutable memory unless persistence is explicitly under test. Validate that every candidate received the same case set.

Make the decision without a fake universal score

Apply hard gates first:

license/runtime/memory compatibility
security invariants
minimum product quality per critical stratum
required tool/schema/context behavior
latency and error SLOs at target load

Eliminate any candidate that fails. For survivors, plot a Pareto table:

Choose the smallest/cheapest operational point that meets all requirements only if that is the declared objective. Otherwise document the explicit product tradeoff and decision owner. Keep a runner-up and rollback criteria.

Prevent common evaluation mistakes

• Do not compare base and instruct models as if formatting were identical.
• Do not evaluate full precision and deploy an untested quantization.
• Do not use public leaderboard scores from a different model revision.
• Do not exclude malformed, empty, refused, timed-out, or OOM outputs.
• Do not report model-judge scores without human calibration.
• Do not mix data used for calibration, prompt tuning, and final testing.
• Do not execute generated code/tools with real authority.
• Do not collapse safety, quality, speed, and memory into one unexplained number.
• Re-run gates after any weight, tokenizer, template, runtime, kernel, or hardware change.

Selection checklist

• Write routes, scope, hard gates, SLOs, and tradeoffs before candidates.
• Verify artifact lineage, license, code, context, and hardware feasibility.
• Combine a small public anchor suite with a protected product set.
• Test tool, structured, multi-turn, safety, and adversarial behavior in the real harness.
• Benchmark the exact served artifact on target hardware and load.
• Pin prompts, schemas, templates, sampling, and evaluator code.
• Use paired case-level statistics and retain failures in denominators.
• Publish a reproducible run bundle and explicit limitations.
• Gate first, then make a documented Pareto decision with rollback criteria.