Do Repository Instruction Files Help Coding Agents? A Benchmark Protocol

The only reliable way to know whether a repository instruction file helps your coding agent is a paired evaluation: run the same held-out tasks from the same repository revisions under controlled instruction conditions, then compare task success, safety, cost, and behavior. Do not infer value from adoption, agent activity, or one successful patch.

This article defines a reproducible protocol. It reports no result because the experiment described here has not been run.

State the decision before designing the benchmark

A useful benchmark answers a concrete release question:

Should this repository ship instruction file version 4 for agent X under harness Y, compared with no file or the current file?

That scope matters. Results for one agent, model, repository, language, or task distribution do not automatically transfer to another.

The February 2026 Evaluating AGENTS.md preprint is the most directly relevant published experiment as of this article's verification date. It compared absent, model-generated, and developer-written context in its defined settings and found effects that varied by file source and evaluation set, while context often increased agent exploration and inference cost. Its authors also identify limits including a Python-heavy task distribution. Use the paper as prior evidence and a methodology reference, not as your repository's answer. Gloaguen et al., Evaluating AGENTS.md

Define four instruction conditions

Prepare the conditions before running held-out tasks:

1. Absent: remove repository-specific agent instruction files from the evaluation checkout.
2. Current human-written: use the exact committed file maintainers currently endorse.
3. Minimal gap-only: retain only commands, constraints, and non-inferable facts required for safe completion.
4. Generated or verbose: create a file using a pinned generator procedure, or use a deliberately comprehensive file representative of the proposal being considered.

Do not call one condition “good” or “bad” in metadata visible to evaluators. Give them opaque IDs.

If you evaluate nested files, define the exact directory scope and verify how the chosen harness resolves precedence. The official AGENTS.md guide says the nearest nested file takes precedence, but this is format guidance; harness implementations still need testing. AGENTS.md official guide

Archive each condition by content digest. Record the generator model, prompt, repository revision, and generation date for any generated file. Never regenerate it between trials.

Build tasks without instruction leakage

Use real repository tasks or maintainer-authored cases that:

• start from a pinned commit before the solution;
• have an independently verified reference behavior;
• include hidden tests or acceptance criteria;
• span bug fixes, features, tests, and refactors relevant to production work;
• vary in which repository-specific facts are necessary;
• exclude changes that created the instruction file itself.

SWE-bench established a repository-level pattern using real GitHub issues and corresponding pull requests, with execution-based evaluation against repository tests. It also documents contamination and reproducibility challenges that matter when mining historical tasks. Jimenez et al., SWE-bench

Create two task strata:

• instruction-relevant: success plausibly depends on a non-obvious command, boundary, or rule in the file;
• instruction-neutral: the file should add little value and may expose overhead.

Have maintainers label relevance before seeing agent outcomes. Keep a sealed test set separate from tasks used to author or prune the instructions.

Hold the agent and environment constant

Instruction condition is the treatment. Keep these fixed:

• agent and harness version;
• returned model identifier and inference configuration;
• system prompt and tool definitions;
• container image, operating system, CPU and memory limits;
• network policy and dependency cache;
• repository commit and task prompt;
• token, time, model-call, and tool-call budgets;
• retry and stopping rules;
• evaluator and hidden tests.

Run each trial in a fresh workspace. Do not let a prior condition leave build outputs, memories, caches keyed by instruction content, or repaired dependencies for the next condition.

Use the same allowed tools. If one condition causes the agent to discover a command that another misses, that is a legitimate treatment effect. If only one condition is given a preinstalled dependency, it is an environment confound.

Randomize and pair trials

For every task, run all conditions. Randomize their order and distribute them across equivalent workers and time windows. Repeat trials because model-driven agents can vary even with identical inputs.

interface TrialManifest {
  readonly taskId: string;
  readonly conditionId: string;
  readonly repositoryCommit: string;
  readonly instructionDigest?: string;
  readonly harnessVersion: string;
  readonly modelId: string;
  readonly environmentDigest: string;
  readonly budget: {
    readonly deadlineMs: number;
    readonly maxModelCalls: number;
    readonly maxToolCalls: number;
    readonly maxTokens: number;
  };
}

Pair analysis by task so repository difficulty does not masquerade as an instruction effect. If budget limits prevent full replication, publish the missing cells and rationale rather than silently comparing different task sets.

Measure outcome, safety, and behavior

Use task completion as the primary outcome:

• hidden tests and acceptance criteria pass;
• patch applies cleanly to the starting revision;
• no unrelated behavior regresses;
• required files and public interfaces are correct.

Add secondary measures:

Do not use “number of tests run” as a standalone quality metric. More activity can be useful, wasteful, or both. Connect behavior to successful outcomes.

The site's agent evaluation guide explains case versioning and release gates, while statistical confidence for agent evals covers paired uncertainty and flaky trials.

Evaluate instruction adherence separately

A task can succeed while ignoring an instruction, or fail despite following every instruction. Score adherence at the rule level:

interface InstructionObservation {
  readonly trialId: string;
  readonly ruleId: string;
  readonly applicable: boolean;
  readonly outcome:
    | "followed"
    | "violated"
    | "not_observable"
    | "not_applicable";
  readonly evidenceIds: readonly string[];
}

Give stable IDs to rules before the experiment. Evidence might be a command event, patch line, test result, or policy denial. Do not ask an LLM judge to infer compliance from the final prose alone.

This analysis answers two different questions:

1. Does the agent respond to the file?
2. Does that response improve the product outcome?

The 2026 study observed that agents often followed instructions and explored more even when task-success effects were limited. That is exactly why adherence and utility must remain separate. Study trace analysis

Prevent evaluator leakage

The evaluator must not receive:

• the condition's descriptive label;
• the reference patch;
• hidden-test expected outputs beyond execution results;
• another condition's trajectory;
• a summary that says which file is expected to win.

Use deterministic checks first. For human or model review, present normalized diffs in randomized order. Calibrate rubric judgments with multiple reviewers and preserve disagreements.

Avoid evaluating with a model that generated one condition unless that dependency is disclosed and analyzed. It may introduce style or instruction-affinity effects.

Analyze a constrained decision

Before looking at results, define gates such as:

• no increase in critical policy violations;
• task success is not worse beyond an agreed tolerance;
• median and tail resource use fit product budgets;
• instruction-relevant tasks show adequate rule adherence;
• no repository subgroup has unacceptable regressions.

Report per-task outcomes and uncertainty, not only an aggregate score. A small average improvement can conceal severe regressions in database or security changes.

Do not combine cost, quality, and safety into one score unless stakeholders approved the weights before the run. Eliminate conditions that fail hard safety gates, then compare the remaining quality-cost frontier.

Diagnose why a condition changed behavior

Review traces for:

• time before the first relevant edit;
• repeated repository traversal;
• commands copied directly from the file;
• conflicts with README or package scripts;
• attention spent on irrelevant architecture;
• tests selected or skipped;
• context-window pressure;
• failures caused by stale commands.

Use the diagnosis to remove or sharpen individual rules. Then create a new instruction version and re-run on an untouched test set. Do not edit the file until it performs well on the same held-out cases; that turns the test set into training data.

Reproducibility checklist

• Write the release decision and gates before running trials.
• Digest and archive every instruction condition.
• Pin repository, agent, model, tools, environment, and budgets.
• Use fresh workspaces and prevent cross-condition memory.
• Pair every condition on the same tasks and randomize order.
• Separate instruction-relevant and neutral strata.
• Score task success, safety, resources, and adherence separately.
• Blind reviewers to condition labels and reference solutions.
• Publish missing trials, failures, and per-task outcomes.
• Reserve a new held-out set for the next revision.