Build an Evidence-First Code Review Agent

An effective code review agent should report only findings it can tie to a changed line, a violated invariant, and verifiable evidence. Pin the exact base and head commits, run deterministic checks first, investigate the diff in repository context, and emit a small typed finding with a reproducer or proof. Leave approval and risk ownership to humans.

This method optimizes for useful defects found per reviewer minute, not comment volume. It does not claim that a model can prove code correct. GitHub's review model explicitly separates comments, approval, and requests for changes; an agent should contribute evidence without impersonating the accountable reviewer. See GitHub's pull request review documentation.

Freeze the review object

Never review “the current pull request” as a moving target. Resolve and record:

• repository identifier;
• full base commit;
• full head commit;
• merge base, if the workflow uses one;
• diff-generation options;
• repository instruction and policy version;
• verifier environment;
• agent, model, and prompt version.

Generate the patch independently with Git. git diff documents comparisons between commits, path filters, rename detection, --check, and machine-oriented output options. Choose one comparison rule that matches the hosting platform, document it, and use the same rule for the agent and final review UI.

If the head changes, cancel or mark the review obsolete. A finding against old content may now point to the wrong line or describe code that no longer exists.

Run deterministic evidence producers first

Compilers, linters, tests, schema validators, dependency scanners, and generated-file checks usually provide stronger evidence than model inference. Run the repository's trusted checks before asking the agent to reason.

Give the agent:

• the bounded diff;
• relevant surrounding symbols and tests;
• concise check results with artifact links;
• ownership and risk metadata;
• repository instructions;
• explicit output schema.

Do not flood context with the entire repository or megabytes of logs. Let the agent request narrow reads through audited tools. Every requested path must remain inside the checkout and every command must run under the coding-agent harness.

Repository text is untrusted. A source comment, fixture, or changed instruction file can contain prompt injection. It may explain intended behavior, but it cannot grant network access, reveal secrets, suppress policy, or authorize publication.

Define a strict finding contract

A review finding should be small enough to verify without rerunning the model:

type ReviewFinding = {
  ruleId: string;
  title: string;
  severity: "blocking" | "high" | "medium" | "low";
  confidence: "high" | "medium";
  file: string;
  line: number;
  changedLine: boolean;
  invariant: string;
  evidence: string;
  reproducer?: {
    command: string;
    expected: string;
    observed: string;
  };
  impact: string;
  suggestedDirection?: string;
};

Require a changed line unless the finding concerns a direct consequence of the change at a nearby unchanged line. Point to the smallest useful range. The title should state the failure, not a category such as “potential bug.”

Use severity for consequence and confidence for evidentiary strength. A hypothetical catastrophic issue is not automatically a blocking finding. Reject output with low confidence, missing evidence, nonexistent paths, stale line numbers, or style-only preferences.

Make evidence concrete

Strong evidence usually takes one of four forms:

1. Deterministic failure: a trusted command fails and the log identifies the changed behavior.
2. Minimal reproducer: a focused test or input demonstrates observed versus required output.
3. Control-flow proof: the changed path necessarily violates an invariant under stated inputs.
4. Authoritative mismatch: the code contradicts a cited specification, schema, API contract, or repository rule.

“This could cause a race condition” is not evidence. A useful report identifies the shared state, competing operations, missing synchronization, reachable interleaving, and resulting invariant violation.

When the agent proposes a reproducer, run it in a disposable environment. Preserve the exact command, exit code, stdout/stderr artifact, and environment identifier. Never execute a shell fragment copied directly from model prose; parse it through the same allowlist and argument-vector policy as any other tool call.

Do not let the agent change the submitted patch while reviewing it. Review and repair are separate states. Otherwise the evidence no longer describes the artifact a human is evaluating.

Review behavior, not aesthetics

Prioritize defects that affect:

• correctness and data integrity;
• authentication, authorization, and secret exposure;
• backward compatibility and migrations;
• resource leaks, unbounded work, or destructive retries;
• concurrency and ordering;
• accessibility and user-critical flows;
• missing tests for a new failure boundary.

Suppress formatting, naming, and import-order comments when automated tools can decide them. Do not request broad refactors unless the change introduces a concrete defect that cannot be fixed locally. A review agent that rewrites design taste into dozens of comments trains developers to ignore it.

Deduplicate findings by root cause. If one missing authorization check affects five endpoints generated by the same helper, report the shared defect and representative locations rather than five nearly identical comments.

Investigate with a hypothesis loop

For each suspected issue, require this sequence:

1. State the invariant that should hold.
2. Identify the changed operation that may violate it.
3. Trace only the necessary callers, callees, types, and tests.
4. Search for an existing guard or repository convention.
5. Attempt a deterministic reproducer or proof.
6. Try to falsify the hypothesis.
7. Emit a finding only if material evidence remains.

The falsification step matters. Search for validation performed by a caller, framework guarantee, database constraint, feature flag, or test fixture that makes the suspect path unreachable. Cite the exact code or primary documentation when such a guarantee is central.

Set per-finding tool and time budgets. If the agent cannot complete the proof, it may return an internal “unresolved hypothesis” for telemetry, but that should not become a developer-facing comment.

Add risk-aware routing

Use changed-path and semantic signals to decide which specialist checks run. Examples:

Routing should be deterministic policy, not a model's decision about whether scrutiny is necessary. GitHub's CODEOWNERS documentation explains how repository paths can request designated reviewers. Treat ownership as one input; sensitive semantic changes may need additional reviewers even when paths look ordinary.

Keep humans in the approval path

The agent should never approve its own generated change. A human or separately governed control decides whether evidence is sufficient and whether residual risk is acceptable.

Present findings in severity order, but make each one independently dismissible. Record human outcomes:

• accepted and fixed;
• accepted, risk consciously retained;
• false positive;
• duplicate;
• stale after update;
• not actionable.

Use those labels to measure the system, not to silently train on private code without documented authorization from the code owner. Useful metrics include confirmed finding rate, blocking-defect recall on a curated corpus, duplicate rate, stale-comment rate, time to first valid finding, verification cost, and reviewer time saved. Report uncertainty and corpus composition; do not optimize only for comment acceptance, which reviewers can inflate by accepting harmless suggestions.

Evaluate before enabling automatic comments

Build a versioned set of real, redacted historical defects and carefully constructed negatives. For each case, store commits, expected invariant, evidence, severity range, and allowed alternative explanations.

Compare the agent with deterministic checks and a no-agent baseline. Evaluate:

• whether it identifies the root cause;
• whether file and line resolve at the pinned head;
• whether the reproducer works;
• whether severity is calibrated;
• whether clean changes remain quiet;
• whether malicious repository text changes policy behavior.

Start by saving reports as private artifacts. Then expose them to reviewers in a dashboard. Enable automatic pull-request comments only after noise and security behavior are acceptable, and keep a kill switch.

Common failure modes

Diff-only tunnel vision: the agent misses a caller, type, or invariant. Allow targeted context retrieval.

Whole-repository speculation: broad context produces unrelated findings. Bound investigation to hypotheses caused by the patch.

Invented execution: the report says a test failed when it was never run. Separate proposed reproducers from verified results in the schema.

Line drift: comments attach to old code. Bind every finding to the head commit and discard stale output.

Instruction injection: changed files tell the reviewer to ignore policy. Treat repository content as data and keep privileges outside the model.

Self-healing review: the agent edits tests until its hypothesis passes. Keep reviewer workspaces read-only except for an isolated scratch reproducer.

Publication checklist

• Base, head, comparison rule, policy, and model version are recorded.
• Deterministic checks run before model review.
• Repository content cannot change privileges or review policy.
• Every published finding names an invariant and concrete evidence.
• Proposed and executed reproducers are visibly different.
• Findings resolve to the pinned head and preferably a changed line.
• Style and duplicate comments are suppressed.
• Sensitive changes route to accountable owners.
• The agent cannot approve or modify the patch it reviews.
• False-positive, staleness, cost, and human-outcome metrics are monitored.