Tail Sampling and Redaction for AI Agent Traces

For AI-agent traces, minimize data before collection, redact allowed telemetry before export, and tail-sample only the sanitized trace. Tail sampling can retain rare errors and slow workflows, but it is not a privacy control: the collector must receive and buffer a trace before deciding, and sampled-out sensitive data was still processed.

This article focuses on collector policy. For span creation and context propagation, start with tracing AI agents with OpenTelemetry.

Design the data boundary first

The current OpenTelemetry GenAI convention marks message content, system instructions, tool definitions, and similar fields as opt-in and warns that they may contain sensitive information. Keep them off by default. Prefer metadata that answers operational questions: operation, model identifier, token usage, time to first token, status, tool name, and non-sensitive outcome codes. See the official GenAI span definitions.

OpenTelemetry’s sensitive-data guidance assigns implementers responsibility for minimizing, filtering, redacting, or transforming telemetry. It warns that hashing low-entropy values can be reversible through guessing. Therefore:

1. Do not instrument secrets or full prompts by default.
2. Delete disallowed attributes at the earliest trusted boundary.
3. Use an allowlist for exported attributes.
4. Keep diagnostic content in a separately approved path, if it exists at all.
5. Restrict access, retention, and downstream copies independently of sampling.

Redaction cannot reliably recover secrets embedded inside arbitrary generated prose. Preventing capture is stronger than regex cleanup.

Put processors in a fail-safe order

A conceptual pipeline is:

receivers
  -> memory controls
  -> resource/attribute normalization
  -> redaction or transformation
  -> tail sampling
  -> batching
  -> approved exporters

The exact order depends on collector components and load, but two invariants should hold: no unredacted data reaches an exporter, and tail-sampling policy uses attributes that have not been removed unexpectedly. If sampling needs a sensitive raw value, redesign the signal—emit a trusted boolean or low-cardinality classification at the application instead.

OpenTelemetry lists both the redaction processor and tail-sampling processor as beta for traces at this review date. Pin collector and component versions, validate configuration against that release, and canary upgrades.

Redact with an allowlist

The collector-contrib redaction processor supports allowed_keys, blocked key patterns, and blocked value patterns. An allowlist is easier to reason about than trying to enumerate every secret name an application might emit.

Example principles—not a copy-paste production config:

processors:
  redaction/agents:
    allow_all_keys: false
    allowed_keys:
      - service.name
      - service.version
      - gen_ai.operation.name
      - gen_ai.request.model
      - gen_ai.response.model
      - gen_ai.usage.input_tokens
      - gen_ai.usage.output_tokens
      - error.type
    blocked_values:
      - '(?i)bearer\s+[a-z0-9._-]+'

Verify names against the component version you deploy. Empty or mistaken allowlists can remove essential telemetry; broad allowlists can leak it. The processor documentation warns that ignored keys bypass checks and therefore require trust. It also supports summary modes; a diagnostic summary that lists redacted key names can reveal schema information, so use silent behavior where names themselves are sensitive.

If correlation requires a stable pseudonym, prefer a keyed HMAC under a rotated, access-controlled key to an unsalted hash. Still treat the result as personal data when it can single out a user. Do not put raw identity in W3C Baggage; the Baggage Recommendation notes privacy and header-abuse risks and calls for removal at trust boundaries.

Tail-sample for explicit operational questions

Head sampling decides before the trace finishes. Tail sampling buffers spans and decides after observing outcomes, which enables policies such as:

• keep errors or selected status codes;
• keep traces above a latency threshold;
• keep a small probabilistic baseline of successful traffic;
• keep rare, trusted workflow classifications;
• drop known health checks or synthetic noise;
• cap retained traces with rate limiting.

Avoid one giant boolean expression nobody can audit. Define policy precedence and expected volume. A practical strategy might always retain severe internal errors, retain a bounded fraction of slow traces, and sample normal successes probabilistically. Safety events may require a distinct, access-controlled incident pipeline rather than ordinary observability retention.

Sampling decisions should use low-cardinality attributes emitted by trusted code. A prompt must not be able to set incident.severity=critical and force expensive retention.

Route every trace consistently

The tail-sampling processor documentation states that all spans for a trace must arrive at the same collector instance. Place a trace-aware load-balancing layer before the sampling tier, keyed by trace ID. Ordinary round-robin distribution fragments traces and makes decisions incomplete.

Late spans are another design choice. The processor exposes decision caches and timing controls, but a span arriving after a decision can be treated differently depending on configuration. Choose decision_wait, trace capacity, and late-span behavior from measured trace durations. Long-running agents may outlive defaults designed for short web requests.

Account for:

• maximum concurrent traces;
• spans per trace and unusually large swarms;
• memory used during the decision window;
• collector restart and resharding behavior;
• spans that arrive after cancellation;
• downstream backpressure and exporter failure.

Tail sampling is stateful. Load-test it under burst, fan-out, and long-duration traces rather than sizing it from average span rate.

Preserve trace meaning after redaction

Redaction can make two operations indistinguishable or break dashboards. Replace removed data with safe classifications at the source:

• agent.task.class = "support_search" instead of the user query;
• tool.result.class = "not_found" instead of the result body;
• policy.outcome = "blocked" instead of the policy evidence;
• document snapshot IDs instead of retrieved text;
• prompt template version instead of system instructions.

Do not synthesize a conversation ID from a trace ID or content hash; the GenAI convention explicitly says to record a real conversation identifier only when one exists. Keep trace IDs for trace correlation.

Metrics derived from sampled traces are generally biased by policy. A dataset that retains every error and few successes cannot directly estimate production error rate. Produce service-level counters and histograms before tail sampling or from a separate metrics pipeline.

Test redaction as a security control

Create adversarial fixtures containing secrets in:

• attribute keys and values;
• resource attributes;
• event attributes;
• log bodies correlated with the trace;
• tool names and error messages;
• Unicode, case, and encoded variants;
• oversized values and nested structures.

Send fixtures through the actual collector build and inspect every exporter, retry queue, debug endpoint, and collector log. Confirm that blocked data does not appear when an exporter fails. Test config parse failures and processor crashes: a telemetry pipeline should fail closed for disallowed destinations, not bypass redaction to preserve availability.

Redaction processor coverage is not universal for every signal and nested body. Read its exact version documentation and add transformations or application-side suppression for unsupported fields.

Validate sampling with known populations

Generate traces with controlled outcomes and verify:

1. Every required error class is retained within capacity policy.
2. Slow-trace thresholds use the intended root duration.
3. Success sampling approximates the configured probability over sufficient volume.
4. Rate limits cap retained volume.
5. Multi-collector routing keeps each trace intact.
6. Long and late traces behave as documented.
7. Redaction occurs even for sampled-out traces and exporter retries.
8. Collector memory remains within limits during bursts.

Monitor spans received/exported, traces sampled by policy, decision latency, incomplete traces, late spans, refused data, redaction counts without sensitive values, exporter failures, and collector memory. Alert when a policy suddenly retains far more or less data than expected.