An AI-agent SLO should measure whether valid user tasks reach a correct, policy-compliant outcome within an agreed time. Model API availability is only a component indicator. Define the event, denominator, workload class, measurement method, target, and window; then maintain separate objectives for reliability, latency, quality, safety, and cost where combining them would hide harm.
Start from the user journey
Google’s SRE guidance on service-level objectives defines an SLI as a quantitative measure and an SLO as a target value or range. It recommends starting from what users care about rather than what is easiest to measure.
For an agent, map the journey:
request accepted -> plan -> model/tools -> verified outcome -> user receives result
A user does not receive a good outcome if the model responds but a required tool fails, the answer lacks evidence, a write is duplicated, or the response arrives after it is useful. Instrument the root task and verify downstream state.
Separate workload classes. Interactive search, background code migration, and high-risk account changes have different latency and correctness needs. One blended objective can allow abundant easy tasks to hide failures in a critical minority.
Define “good” and “valid” precisely
The Google SRE workbook uses the pattern of good events divided by valid events. Adapt it carefully:
task-success SLI =
valid tasks with verified acceptable outcome
------------------------------------------------
all valid tasks
Write the denominator policy before measuring. Define treatment of malformed requests, policy-blocked requests, user cancellations, test traffic, provider outages, and tasks the product does not claim to support. Exclusions should be observable, version-controlled, and reviewed; otherwise a team can improve the SLO by reclassifying failures.
Define good outcomes with independent evidence:
- deterministic tests or schema validation;
- verified tool state and idempotency;
- cited-source checks where facts require sources;
- calibrated human or judge rubrics for subjective work;
- explicit policy and authorization results.
The agent’s own statement that it succeeded is not an SLI.
Use a small portfolio of objectives
Do not compress everything into one weighted score. A high average quality score can offset a severe safety failure mathematically while remaining unacceptable operationally.
Reliability
Measure the proportion of valid tasks reaching a terminal acceptable state. Distinguish safe, explained failure from timeout, wrong success, and unknown side-effect state. Depending on the product promise, a fast truthful refusal may be a correctly handled event, but it is not a successful completion.
Latency
Measure time to first useful output and time to final outcome for each workload class. State percentile, target, and window. Queueing and retries count from the user’s start time. Use the process in our agent latency-budget guide.
Quality
Measure task-specific correctness, groundedness, instruction adherence, or usefulness. Deterministic checks can cover every event; expensive human review may cover a stratified sample. Preserve critical slices rather than reporting only one mean.
Safety and policy compliance
Track unauthorized side effects, sensitive-data exposure, policy violations, and control bypasses independently. Do not promise “zero risk”; define event severity, incident response, and the strictest measurable objective the system can operate against. Some severe events warrant immediate response regardless of a rolling budget.
Cost efficiency
Track direct and allocated cost per verified successful outcome, plus high-percentile cost and budget-exceeded rate. A cost objective must not cause silent quality or safety degradation. Our multi-agent cost-attribution guide provides the ledger model.
Handle delayed and sampled quality labels
Unlike HTTP status, outcome quality may be known hours or days later. Maintain two views:
- Provisional SLI: immediately available deterministic and automated checks.
- Matured SLI: the same event cohort after its label window closes and sampled reviews arrive.
Version evaluator prompts, models, rubrics, and calibration sets. Save the evaluator version on each label. When a label is corrected, append the revision and recompute the affected cohort; do not silently overwrite history.
Sampling must represent task classes and deliberately oversample rare critical slices. Apply documented weights when estimating a production-wide rate. Report sample size and uncertainty. A hundred reviewed events cannot support a precise claim about a one-in-ten-thousand failure.
Use agent evals in CI for release prevention, but do not substitute an offline benchmark for the production SLI. Real inputs, tool states, provider behavior, and distribution shift differ.
Write an SLO specification a reviewer can reproduce
Leave target values unset until stakeholders choose them from user need and observed capability:
name: interactive_verified_task_success
population: authenticated interactive tasks in supported categories
good_event: terminal success plus tool-state verification and policy pass
valid_event: accepted task excluding documented user cancellations
target: X%
window: rolling N days
label_maturity: 72 hours
owner: agent-platform
data_source: root workflow events + verification ledger
exclusions_version: git:...
Also document late-arriving events, duplicate task IDs, telemetry loss, regional aggregation, maintenance, and rounding. If missing telemetry is ignored, an observability outage can appear as perfect reliability. Prefer treating unknown events conservatively or exposing a separate coverage objective.
Test the query against a frozen fixture before adoption. The fixture should contain a good task, a verified failure, a cancellation, a duplicate event, an excluded synthetic task, a missing label, and a late correction. Have product, reliability, safety, and finance owners calculate the same numerator and denominator independently; disagreement reveals an underspecified objective before it reaches an incident review.
Turn the target into an error budget
An SLO below 100% implies an allowed amount of not-good service in the window. Google’s example error-budget policy connects budget consumption with release and reliability decisions.
Define a policy with stakeholders:
- who owns the budget;
- which failures consume it;
- what happens at fast and slow burn rates;
- when risky launches pause;
- which safety events bypass budget arithmetic;
- how evaluator revisions affect historical burn;
- how the policy resumes after recovery.
An error budget is not permission to cause failures. It is a shared decision mechanism for balancing change and reliability.
Alert on symptoms and diagnose with components
Page on rapid user-impact burn, unsafe effects, or a large unknown-state increase. Use component metrics—provider errors, tool latency, queue age, retries, token growth, route changes—to diagnose. The Google SRE monitoring workbook distinguishes indicators that show user impact from diagnostic dashboards that explain why.
Avoid paging directly on every model-provider error if fallbacks preserve the user outcome. Conversely, page on task failure even when each provider’s component uptime is within contract. The root journey is the product.
Guard against gaming
Each objective creates incentives. Review these failure modes:
- rejecting difficult valid tasks to improve success rate;
- streaming meaningless text to satisfy first-token latency;
- shortening answers until quality falls;
- excluding provider incidents from the denominator;
- routing only evaluation traffic to the strongest model;
- letting an uncalibrated model judge its own outputs;
- dropping required safety checks to meet a deadline;
- measuring cost per request instead of per success.
Counter them with paired objectives, immutable population rules, hidden audit samples, and independent state verification. Revisit the SLO when product promises or workload distribution changes—not merely when the target becomes inconvenient.