Use the highest-level interface that completely expresses the task: a narrow API first, structured browser automation second, accessibility-tree or DOM grounding next, and screenshot-based computer use when the required information or control is available only visually. Lower-level interfaces increase coverage, but they also increase ambiguity, latency, and the surface for unsafe mistakes.
Compare the interfaces by contract
| Interface | Agent observes | Agent acts with | Best fit | Primary failure |
|---|---|---|---|---|
| Domain API | Typed objects | Validated operations | Stable owned workflows | Missing endpoint or semantic gap |
| Browser DOM | Roles, labels, attributes | Locators and events | Web apps with usable semantics | Dynamic structure or weak markup |
| Accessibility tree | Exposed semantic UI | Role/name-based targets | User-facing semantic controls | Visual-only meaning omitted |
| Screenshot/computer use | Pixels and coordinates | Pointer/keyboard | Canvas, remote desktop, visual-only UI | Ambiguous grounding and layout drift |
These modes can be combined. Selection should happen per step, not once per agent.
Prefer an outcome-oriented API when available
An API can express create_refund_preview(invoice_id) instead of clicking through several screens. The server can validate identity, scope, idempotency, and invariants at one boundary. APIs are also easier to contract-test and observe.
Do not expose a generic HTTP request tool as a shortcut. It can widen network and authorization access beyond the intended outcome. Wrap APIs in narrow tools with bounded schemas and separate read, preview, approve, and commit operations.
Use browser automation when the UI contains necessary policy or behavior absent from the API, or when testing the user journey is itself the goal.
Use semantic browser targets before coordinates
Playwright recommends user-facing attributes such as role and accessible name for resilient locators, and its locators re-resolve the target before each action. See the Playwright locator documentation.
await page.getByRole("button", { name: "Preview refund" }).click();
await expect(page.getByRole("heading", { name: "Review refund" })).toBeVisible();
This states intent more clearly than clicking (642, 511). It also gives the application a chance to fail when two controls have the same ambiguous name rather than choosing one silently.
DOM access is not inherently trustworthy. Page text and attributes are controlled by the site and may contain prompt injection. The automation layer should decide allowed origins, actions, downloads, and data transfer independently from the model’s interpretation.
Use the accessibility tree as a complementary view
The accessibility tree exposes roles, names, values, states, and hierarchy derived from native semantics and ARIA. It can remove visual noise while showing what assistive technology receives. Playwright’s ARIA snapshots provide a structured representation suitable for assertions. See Playwright ARIA snapshots.
The tree can still be incomplete or wrong when the page is poorly implemented, and it omits some pixel-only meaning. Never assume an aria-label proves the visible control matches its description on an untrusted page.
Reserve screenshots for irreducibly visual steps
Screenshots are necessary for charts, canvases, image editors, remote desktops, maps, spatial arrangement, and interfaces that expose no usable semantics. When using them:
- capture viewport size, device scale, scroll position, and image hash;
- pair the image with DOM/accessibility evidence when available;
- require a confidence threshold or confirmation for ambiguous targets;
- verify the postcondition after every meaningful action;
- never reuse coordinates after navigation or layout change; and
- crop sensitive areas before sending pixels outside the trust boundary.
VisualWebArena was introduced to test visually grounded tasks that text-only web benchmarks omit. Its existence supports evaluating visual capability separately; it does not show that screenshot control is preferable for semantic web tasks. See VisualWebArena.
Route each step with a decision function
type InterfaceChoice = "api" | "dom" | "a11y" | "vision";
function chooseInterface(step: Step): InterfaceChoice {
if (step.hasNarrowAuthorizedApi) return "api";
if (step.hasStableSemanticLocator) return "dom";
if (step.exposedInAccessibilityTree) return "a11y";
return "vision";
}
This is a priority policy, not sufficient authorization. Add risk, ownership, and verification requirements. A screenshot-only “Delete” button should not become safe because vision can locate it.
Verify outcomes out of band
After an action, inspect authoritative state through the narrowest available path. A toast saying “Payment sent” is weaker evidence than the transaction API returning the new record. For browser-only workflows, assert URL, semantic page state, and a stable record identifier.
Preview irreversible operations and require explicit approval that includes target, values, external consequence, and expiry. Bind the approval to an action digest so layout changes or prompt text cannot broaden it.
Benchmark the routing policy
Use the same task set to compare API-only, semantic-browser, visual-only, and hybrid agents. Measure task success, unintended effects, steps, latency, model tokens, screenshots processed, intervention, and postcondition verification. Stratify by visual dependency.
WebArena provides self-hosted, programmatically evaluated web tasks across several realistic site categories and emphasizes functional correctness. Its structure is a useful model for an internal benchmark where effects can be reset. See WebArena.