Statistical Confidence for AI Agent Evaluations

Statistical confidence in an agent evaluation begins by defining what population and randomness the result represents. Report counts and uncertainty, pair baseline and candidate on the same cases, preserve task-family dependence when resampling, and distinguish “no detected regression” from evidence of acceptable performance.

Statistics cannot rescue biased cases, invalid graders, leaked test data, or missing safety checks. Build those foundations first in the agent eval CI guide.

State the estimand before choosing a formula

An estimand is the quantity you intend to estimate. Two common questions differ:

1. Fixed-suite performance: What proportion of this exact, versioned set did the configuration pass under this trial protocol?
2. Generalized performance: What success rate should we expect on a broader population of similar future tasks and conditions?

The first can be a complete census of a fixed suite; randomness may come only from repeated model/tool executions. The second treats cases as sampled from a population and requires defensible sampling and a model of case variation.

NIST AI 800-3 explicitly distinguishes benchmark accuracy conditioned on a fixed benchmark from generalized accuracy over similar potential items and shows that unexamined statistical assumptions can produce invalid uncertainty estimates. Keller et al., NIST AI 800-3

Write down the system, user/task population, time period, configuration, sampling procedure, trial protocol, metric, and aggregation. Without that statement, “95% confidence” has no stable meaning.

Identify the independent unit

Rows are not automatically independent observations. Ten paraphrases of one incident, five turns from one conversation, or repeated trials on one task share causes.

Choose the highest unit sampled independently for the claim:

• customer or workflow family for population coverage;
• task case for a paired system comparison;
• run within task for stochastic reliability; and
• day or deployment window for time-varying production measurements.

If you bootstrap, resample independent groups and keep their dependent rows together. Row-wise resampling of duplicates produces intervals that are too optimistic. For richer designs with task difficulty, repeated generations, judges, and models, use a suitable hierarchical or mixed-effects model with statistical expertise rather than pretending all runs are independent.

Report a single pass rate with an interval

Always show the numerator and denominator. For binomial pass/fail observations under the required independence and common-probability assumptions, a Wilson interval is generally preferable to the simple Wald interval, especially near 0 or 1 or with modest samples. NIST documents Wilson and exact binomial intervals and notes that exact methods are relevant for very small samples or few failures. NIST binomial confidence-interval guidance

Interpret a frequentist 95% confidence interval correctly: the procedure covers the true parameter in about 95% of repeated samples under its assumptions. It does not mean there is a 95% probability that the fixed interval you computed contains the parameter. NIST confidence-limit interpretation

Zero observed failures does not establish zero risk. Report an upper confidence bound on the failure probability, and size high-consequence tests around an acceptable bound before running them.

Compare systems with paired cases

Run baseline and candidate on the same case snapshot and environment. For binary outcomes, summarize the four paired counts:

both pass
baseline pass / candidate fail   ← regressions
baseline fail / candidate pass   ← improvements
both fail

The paired effect is driven by the discordant pairs, not the two marginal pass rates alone. McNemar’s test evaluates whether the two discordant probabilities differ under its assumptions; NIST documents it for paired binary variables and requires independent pairs. NIST McNemar test reference

Also report an effect-size interval for the paired difference and the raw discordant counts. A tiny p-value can describe a trivial change in a huge dataset, while a practically damaging regression can remain statistically inconclusive in a small one.

If every case has repeated stochastic trials, decide whether the primary unit is case success, per-case probability, or individual run. A run-level test that ignores case clustering can overstate precision.

Define reliability metrics carefully

For an agent expected to succeed repeatedly, distinguish:

• pass@k: at least one success in k attempts; and
• pass^k: all k attempts succeed.

They answer opposite operational questions. Best-of-many generation can be appropriate when the product actually runs several attempts and has a valid selector. It exaggerates user-facing reliability when the product runs once. τ-bench introduced pass^k to examine consistency across repeated trials; use its definition only when your protocol matches. Yao et al., τ-bench

Record retry and selection policy. Retries are part of the system, not free independent evidence.

Use bootstrap intervals with the right resampling plan

Bootstrap methods can estimate uncertainty for statistics without convenient analytic formulas, including medians and composite metrics. NIST describes resampling with replacement and cautions that bootstrap methods are not suitable for every distribution or tail-sensitive statistic. NIST bootstrap guidance

Predeclare:

resampling unit: workflow family
stratification: risk tier and customer segment
within-unit handling: keep all cases and trials together
statistic: candidate minus baseline macro pass rate
replicates and random seed: recorded
interval method: named in report

Resample paired baseline/candidate observations together. Preserve sampling weights when estimating production-distribution performance. Do not use a bootstrap to claim representativeness the dataset never had.

Plan sample size around a decision

Before evaluation, specify:

• baseline rate or plausible range;
• smallest regression or improvement that matters;
• false-positive tolerance;
• desired power to detect that effect;
• pairing or clustering structure; and
• expected missing/invalid-run rate.

Then calculate sample size for the selected test or simulate power using realistic per-case variance and correlation. NIST’s guidance shows that sample-size calculations for proportions depend on significance, power, baseline and alternative proportions. NIST sample-size guidance

Do not choose trial count solely from budget and later call an inconclusive result “no regression.” When cost is constrained, report the minimum effect the design could reliably detect.

Control multiplicity and analysis flexibility

Agent reports often compare many models across dozens of tags, metrics, judges, and checkpoints. Searching all of them and announcing the best p-value inflates false positives.

Declare one or a small number of primary outcomes. Treat subgroup results as diagnostics unless powered and predeclared. Apply an appropriate multiple-comparison procedure when making simultaneous claims; NIST lists methods including Bonferroni for preselected contrasts. NIST multiple-comparison guidance

Preserve the analysis plan, excluded cases, stopping rule, and all measured outcomes. Sequential peeking and stopping when a threshold crosses changes error properties unless the procedure was designed for sequential analysis.

Keep safety gates outside average statistics

A candidate can improve mean task success while introducing one cross-tenant read. Define hard policy failures separately and usually require zero observed violations in the tested suite. Still report the number of opportunities and an uncertainty bound; “0/40” is weaker evidence than “0/40,000” under comparable sampling.

For ordinary quality, define a non-inferiority margin from product consequence. A useful release statement is:

The candidate is accepted only if the lower confidence bound for
(candidate − baseline) exceeds −δ on the primary paired metric,
all critical-tag margins pass, and no hard safety case fails.

Choose δ before seeing the results. This is a decision rule, not a universal statistical standard.

Reporting checklist

• State fixed-suite or generalized estimand and target population.
• Identify independent sampling and resampling units.
• Report numerators, denominators, missing runs, and intervals.
• Pair baseline and candidate by case and preserve clustering.
• Name interval, test, assumptions, and software implementation.
• Report practical effect sizes, not p-values alone.
• Predeclare sample size, margins, primary metrics, and stopping rules.
• Adjust or limit simultaneous claims.
• Keep hard safety failures separate from averages.
• Publish dataset, configuration, and analysis versions where permitted.