To bisect an agent regression, first make the failure classifiable under a frozen evaluation environment, then search one ordered change axis at a time. A stochastic one-run pass/fail predicate is not suitable for binary search: use repeated trials and a predeclared “good, bad, or inconclusive” rule, and skip commits that cannot be classified.
Confirm that a regression exists
Before searching history, capture:
case and dataset revision
last accepted and first suspect configuration
model identifier and decoding settings
prompt, tool schema, policy, retrieval, and code digests
initial fixture and external-response snapshot
baseline and candidate trial results
first observable failure in the trajectory
Re-run baseline and candidate on the same cases and fixtures. Pair trials where possible. If the old configuration now fails too, investigate environmental drift, expired fixtures, changed model aliases, or grader drift before blaming a code commit.
Treat the predicate as a measurement instrument, not an oracle. NIST AI 800-3 explains why AI evaluations need explicit uncertainty, validity, and context rather than a score detached from its measurement conditions.
Do not reduce a broad score change to bisection until one stable predicate exists. “Overall score decreased” may combine unrelated improvements and failures. Prefer “case family refund-timeout violates at-most-once effect more often than the release tolerance.”
Freeze everything outside the search axis
Agent behavior depends on several independently versioned surfaces:
| Axis | Freeze while searching another axis |
|---|---|
| Source code | container/dependency lock and build flags |
| Prompt | rendered prompt digest, not filename |
| Model | immutable model/version identifier if available |
| Tools | discovery metadata, schemas, results, and policy |
| Data | eval snapshot, retrieval corpus, memory, fixture |
| Evaluator | deterministic checks, rubric, judge, parser |
Use recorded model and tool responses to bisect deterministic orchestration bugs. Use fresh repeated calls to bisect behavior changes in prompts or model configuration. State which mode you used: recorded replay cannot tell whether the current model distribution improved.
If a provider exposes only a moving alias, you may be unable to reconstruct historical behavior. Report that limitation rather than attributing the regression precisely.
Define a three-way classifier
Git’s automated bisect treats exit code 0 as good, 1–127 except 125 as bad, and 125 as “skip.” Git bisect run documentation
An illustrative wrapper might apply this policy:
GOOD: no hard failure and the paired effect is inside the accepted region
BAD: any hard safety failure, or the paired effect exceeds the regression margin
SKIP: build failure, invalid fixture, external outage, or insufficient evidence
For stochastic quality metrics, define the trial count, effect-size margin, interval method, and maximum reruns before starting. Do not repeatedly sample until a desired classification appears. “Inconclusive” is evidence that the experiment lacks precision, not permission to guess.
Use a deterministic hard failure immediately: one unauthorized write is enough to classify a safety regression under most release policies. For routine success rates, classification should consider practical magnitude and uncertainty, not only a p-value.
Bisect source history safely
Once the predicate is reliable:
git bisect start FIRST_BAD LAST_GOOD
git bisect run ./scripts/classify-agent-regression.sh
Git chooses revisions by binary search and can automate the process with a script. Its documentation also warns that skipped revisions may leave ambiguity about the first bad commit. Git bisect manual
The script should build in a clean, disposable worktree or container; restore fixture snapshots; run the pinned evaluator; emit an artifact; and return only the documented code. Never allow historical code to receive production credentials. Old commits are untrusted execution.
Record git bisect log and preserve per-revision results. End with git bisect reset; do not make destructive workspace cleanup part of the automated classifier.
Search configuration history when Git is not the axis
Prompt registries, model assignments, retrieval indexes, feature flags, and tool descriptions may change outside source control. Export an ordered manifest:
{
"effectiveAt": "2026-07-10T18:42:00Z",
"promptDigest": "sha256:...",
"model": "provider/versioned-id",
"toolSchemaDigest": "sha256:...",
"policyDigest": "sha256:...",
"corpusSnapshot": "corpus-2026-07-10.3"
}
Apply the same midpoint search over that ordered history. If several dimensions changed in one deployment, reconstruct factorial comparisons when feasible: old prompt/new model and new prompt/old model can reveal an interaction that a linear history cannot separate.
Expect non-monotonic behavior
Binary search assumes a boundary: earlier states are good and later states bad for the chosen predicate. Agent changes can violate that assumption. Commit A may introduce a failure, B mask it, and C expose it again. A model/prompt interaction may make only the middle revision bad.
After finding a candidate change:
- test its parent and itself with the full predeclared trials;
- revert only that change on the bad revision, if practical;
- apply it to the good revision, if practical;
- run neighboring revisions to test monotonicity; and
- inspect trajectory differences for a causal mechanism.
A bisect result is a suspect, not proof. The revert/application experiments strengthen attribution.
Diagnose the first causal difference
Diff normalized trajectories side by side:
- retrieved source IDs and ranking;
- rendered instructions and available tools;
- first differing tool selection or argument;
- policy allow/deny result;
- tool error and retry decision;
- terminal-state diff; and
- judge evidence, if semantic grading was involved.
The final answer often differs long after the cause. Focus on the earliest semantic divergence. Use deterministic replay for orchestration and fresh trials for model behavior.
Budget the search before running it
Binary search reduces the number of revisions, but repeated agent trials can still be expensive. Estimate the maximum midpoint count from the revision range, multiply by the predeclared trials and per-run ceiling, and set a total budget. Start with a deterministic reproducer or a high-signal case family; confirm the final suspect with the broader suite. Cache only content-addressed builds and immutable recorded dependencies—never stochastic verdicts whose additional trials are part of the classifier. Preserve inconclusive results instead of paying indefinitely for certainty the design cannot achieve.
Avoid invalid shortcuts
- One run per revision: flaky labels break binary search.
- Changing the judge: the ruler becomes part of the regression.
- Live mutable tools: external drift masquerades as source history.
- Deleting hard cases: restores the score, not the behavior.
- Testing old code with real secrets: converts debugging into incident risk.
- Assuming one culprit: batched changes and interactions can have several causes.
- Tuning thresholds mid-search: biases the boundary toward the desired commit.
Bisection checklist
- Reproduce baseline and candidate on paired, pinned cases.
- Define one failure property and practical regression margin.
- Freeze every dependency outside the searched axis.
- Predeclare trials and good/bad/inconclusive rules.
- Use isolated worktrees or containers with no production credentials.
- Preserve the bisect log and result artifact for every revision.
- Check neighboring revisions and non-monotonicity.
- Confirm causality through revert/application tests and trajectory evidence.
- Add the minimal confirmed failure to the eval dataset.