How to Build a High-Quality Agent Evaluation Dataset

A useful agent evaluation dataset is a versioned sample of decisions your product must make correctly—not a folder of clever prompts. Each case should identify its origin, intended behavior, risk, grading method, and limits. Build it from real task distributions and confirmed failures, then protect a sequestered test split from prompt and model tuning.

This guide covers dataset construction. Use it alongside the agent eval release gate, which explains repeated trials and CI policy.

Define the decision before collecting examples

Start with the release question and the evaluation target. “Is the new agent better?” is underspecified. A defensible target looks like this:

Target: customer-support agent configuration 8f31c2
Context: authenticated US retail customers using chat
Decision: may this candidate replace the production configuration?
Critical outcomes: correct resolution, policy compliance, no unauthorized writes
Out of scope: voice calls, non-US policy, administrator workflows

NIST’s AI Risk Management Framework playbook says test sets, metrics, and TEVV tools should be documented, and warns that non-representative data can produce inaccurate assessments or harmful outcomes. It also recommends involving people who know the context of use. NIST AI RMF Measure playbook

Name the complete system under test: model, instructions, retrieval, memory, tools, authorization middleware, and post-processing. A dataset built for final-answer quality alone cannot support claims about safe side effects.

Use a coverage map, not an intuition pile

Create strata before cases. A support-agent map might include:

Do not force equal proportions unless that matches the question. Maintain at least two views:

• a distributional suite, sampled to estimate routine behavior; and
• a risk suite, deliberately enriched with severe and rare failures.

Report them separately. A risk-enriched pass rate is not an estimate of live success, while a frequency-weighted average can hide a catastrophic but rare failure.

Give every case an auditable record

The following TypeScript is an illustrative storage contract, not a package-specific format:

interface EvalCase {
  readonly id: string;
  readonly revision: number;
  readonly task: string;
  readonly initialFixture: string;
  readonly tags: readonly string[];
  readonly risk: "critical" | "high" | "normal";
  readonly origin: {
    readonly kind: "incident" | "sample" | "synthetic" | "expert";
    readonly reference?: string;
  };
  readonly expected: {
    readonly finalState?: unknown;
    readonly requiredFacts?: readonly string[];
    readonly forbiddenEffects?: readonly string[];
    readonly acceptableOutcomes: readonly string[];
  };
  readonly grader: {
    readonly kind: "deterministic" | "human" | "model-assisted";
    readonly rubricVersion: string;
  };
  readonly privacyReview: string;
}

Keep the expected result separate from the prompt presented to the agent. Otherwise an accidental template join can leak the answer. Store fixture and rubric versions by immutable digest so a later run can reconstruct what was judged.

“Datasheets for Datasets” proposes documenting a dataset’s motivation, composition, collection process, and recommended uses. Apply that idea to evals with a checked-in dataset card covering scope, sources, labeling, exclusions, known skews, privacy, license, and change history. Gebru et al., Datasheets for Datasets

Collect cases from four complementary sources

Production-shaped samples estimate common behavior. Sample by stable units such as user request or completed workflow, not individual messages that remove necessary context. Use a defined time window and document eligibility filters.

Confirmed incidents prevent recurrence. Preserve the causal feature while replacing customer names, credentials, and proprietary text. A sanitized case is a regression test, not evidence of incident frequency.

Domain-expert cases cover policy boundaries and rare high-consequence decisions. Ask experts to supply the governing policy passage and acceptable alternatives, not only one ideal response.

Synthetic and adversarial cases fill explicit gaps. Mark them as synthetic and review them. Generated cases can inherit the generator’s blind spots and must not be presented as representative user traffic.

Deduplicate semantically, not just by exact text. Near-duplicates can make confidence intervals look narrower while measuring the same underlying scenario repeatedly.

Label outcomes, not preferred wording

For tool-using agents, ground truth usually has several parts:

1. the acceptable terminal states;
2. effects that must never occur;
3. policy constraints on the route;
4. facts that must be supported; and
5. conditions requiring clarification or escalation.

Prefer state assertions over exact trajectories when several safe routes work. For example, “order remains unchanged and an escalation ticket exists” is more robust than requiring lookup_order before create_ticket unless order is a policy requirement.

Write the rubric before viewing candidate outputs. Give labelers a “cannot determine” option; forced certainty becomes label noise. Double-label a calibration subset independently, inspect disagreements by tag, and have a domain owner adjudicate policy disputes. Raw agreement is easy to understand, but also publish the disagreement matrix: one summary coefficient can conceal systematic confusion.

Split to prevent evaluation overfitting

Use at least three partitions:

• development: visible cases for local iteration;
• release: stable cases evaluated in CI; and
• sequestered: access-controlled cases used periodically to detect overfitting.

Split by underlying incident, customer, document, template, or workflow family—not merely by row. Paraphrases from one source must stay together. If a retrieval corpus contains the reference answer, document that dependency rather than calling it leakage automatically; it may be the intended system behavior.

Version additions and corrections. Never silently change a case after seeing a release result. Record whether a revision fixes an invalid label, changes product policy, or expands coverage, and rerun the baseline on the new snapshot.

Protect people and proprietary data

Removing names is not always enough: free text can identify people through combinations of facts. Minimize collection, document the purpose, use an explicit permission process where appropriate, restrict access, set retention limits, and test redaction. Before collecting personal or human-subject data, confirm the requirements that apply to the deployment rather than treating this engineering checklist as a legal determination. NIST calls for respecting privacy and intellectual-property rights and following human-subject requirements where applicable. NIST Measure 2.1

Do not send sensitive cases to an external evaluator unless its data handling is approved. Keep secrets as fixture handles, never literal credentials. Treat retrieved documents and tool outputs as untrusted test inputs.

Check dataset health on every change

Automate structural checks:

- IDs and revisions are unique
- every case has an origin and privacy review
- referenced fixtures and policy versions exist
- critical cases have deterministic effect checks
- no secrets or forbidden personal-data patterns are present
- train/development families do not overlap sequestered families
- tags and coverage counts match the dataset card

Then review semantic drift on a schedule. Products, policies, tools, and attackers change. Retire obsolete cases transparently; retain their history so old scores remain interpretable.

Publication checklist

• Define the system, population, context, and release decision.
• Maintain separate distributional and risk-focused reporting.
• Attach provenance, revision, fixtures, rubric, and privacy status to every case.
• Prefer terminal-state and safety assertions over brittle prose matching.
• Independently label and adjudicate a calibration subset.
• Group related examples before splitting.
• Keep a sequestered set and audit access to it.
• Publish coverage by tag and known limitations, not only an overall score.
• Add confirmed failures without erasing the historical baseline.