Ground Computer-Use Agents with Screenshots and Accessibility Trees

Screenshots tell a computer-use agent what is visible; accessibility trees tell it which semantic controls the platform exposes. Use both when possible, but never assume they agree. Build candidate targets from the semantic tree, associate them with current geometry and pixels, detect conflicts, and verify the resulting application state after every action.

Understand what each representation omits

WAI-ARIA defines roles and properties exposed to accessibility APIs, but authors can misuse ARIA. Native semantics are generally preferable, and a semantic label from an untrusted page is still untrusted content. See WAI-ARIA 1.2.

Capture one coherent observation

An observation should identify exact browser state:

type GroundedObservation = {
  navigationId: string;
  screenshotHash: string;
  viewport: { width: number; height: number; scale: number };
  scrollX: number;
  scrollY: number;
  semanticNodes: readonly SemanticNode[];
  capturedAt: number;
};

type SemanticNode = {
  id: string;
  role: string;
  name: string;
  states: readonly string[];
  bounds?: { x: number; y: number; width: number; height: number };
};

Browser APIs do not necessarily expose a stable accessibility-node identifier across updates. Scope IDs to the captured observation and re-resolve targets immediately before action.

Pause animations or wait for a declared stable state during evaluation, but do not assume production pages become static. Reject an action when navigation, resize, scroll, or a material DOM mutation invalidated its observation.

Generate and reconcile candidate targets

For the instruction “Open billing settings”:

1. find semantic candidates with compatible role and accessible name;
2. obtain current bounds through the automation layer;
3. inspect the corresponding screenshot crops;
4. check visibility, occlusion, enabled state, and uniqueness;
5. prefer a semantic action when the match is strong; and
6. use coordinate action only when visual meaning is required.

Do not ask a model to transcribe the entire screenshot before considering semantic evidence. Supply compact candidates plus targeted crops to reduce irrelevant visual context.

Treat disagreement as a first-class state

Examples include:

• tree says “Delete project,” visible text says “Archive”;
• button is semantically enabled but visually covered by a dialog;
• chart contains required evidence but has no semantic representation;
• two nodes share one accessible name;
• a transparent overlay intercepts pointer events; or
• a canvas visually resembles a standard input but is not one.

For low-risk navigation, refresh the observation or choose an unambiguous alternative. For external or destructive effects, stop and ask for confirmation with a preview. Never average conflicting evidence into confidence.

Build disagreement metrics: frequency by application, percentage resolved by refresh, wrong-action rate, and cases requiring human intervention. These findings can also identify real accessibility defects.

Use semantic actions when semantics are sufficient

Playwright role locators target how users and assistive technology perceive controls, and locators re-resolve the DOM element for an action. See Playwright locators. ARIA snapshots can assert selected portions of an accessibility tree and are order-sensitive by default. See Playwright snapshot testing.

const target = page.getByRole("button", { name: "Save settings" });
await expect(target).toBeVisible();
await target.click();
await expect(page.getByRole("status")).toHaveText("Settings saved");

The visible status remains an application assertion. For consequential actions, verify the authoritative backend state too.

Use pixel actions with stale-state protection

Bind coordinate actions to screenshotHash and navigation ID. Before dispatch:

• confirm the observation is current;
• verify the point is inside the intended target bounds;
• take a fresh crop for high-risk controls;
• ensure no modal or overlay appeared; and
• record the exact input event.

Afterward, capture a new observation and test a postcondition. If it fails, do not blindly click again; duplicate submission is a common harmful retry.

Account for frames, portals, and shadow boundaries

Target identity must include its browsing context. Two frames can expose controls with the same role and name, while a cross-origin frame may prevent the automation layer from reading DOM details even though pixels remain visible. Record frame origin and hierarchy, constrain which origins the agent may enter, and require a fresh authorization decision before transferring data across them. Shadow DOM and custom elements can also separate visible and semantic structure; test the actual browser accessibility exposure instead of assuming the component implementation is discoverable.

Build accessible agent output too

Grounding from accessibility semantics does not make the agent’s own UI accessible. Streamed plans and tool results need semantic headings, keyboard navigation, predictable focus, and live-region behavior that avoids announcing every token. Approval dialogs need a meaningful accessible name, consequence summary, and clear cancel path.

Use semantic HTML before ARIA. Test with keyboard and representative screen readers; an accessibility-tree snapshot catches structural regression but does not replace assistive-technology testing.

Evaluate fusion against baselines

Compare screenshot-only, semantic-only, and fused observations on tasks stratified by visual dependency. Measure functional success, unsafe effects, grounding errors, steps, latency, context size, disagreement, and accessibility defects encountered. Preserve screenshots and semantic snapshots with privacy controls for audit.

VisualWebArena motivates evaluating tasks where visual grounding is necessary, while WebArena emphasizes functional correctness in resettable sites. Together they support a benchmark that separates observation capability from actual outcome. See VisualWebArena and WebArena.