Trace AI Agents with OpenTelemetry: Spans, Tokens, Tools, and Redaction

An agent trace should answer: what goal was attempted, which model and tools ran, where time and tokens went, what failed, and which user-visible outcome resulted?

OpenTelemetry gives you vendor-neutral traces and metrics for that job. Its GenAI semantic conventions now define common agent, model, and tool attributes, but those conventions are still developing. Instrument behind a small internal layer so a naming revision does not spread across your application.

Draw the trace before writing instrumentation

Use one end-to-end trace for one user-visible workflow. A practical multi-agent topology looks like this:

invoke_agent coordinator
  ├── chat model
  ├── invoke_agent researcher
  │     ├── chat model
  │     └── execute_tool search_docs
  ├── invoke_agent reviewer
  │     └── chat model
  └── execute_tool publish_preview

The coordinator span is the parent of specialist-agent work. Model calls, retrieval, and tools are children of the agent that initiated them. Context propagation ties asynchronous branches to the same trace even when they run concurrently.

Do not model every internal function as a span. Create spans for remote calls, model invocations, tool execution, durable state transitions, and significant orchestration steps. Too many low-value spans raise cost and obscure the critical path.

The official OpenTelemetry GenAI observability guidance recommends this agent-to-model-to-tool trace hierarchy and explicitly warns that captured content can be large and sensitive.

Set up the Node SDK once

The current packages checked for this article are @opentelemetry/api 1.9.1, @opentelemetry/sdk-node 0.220.0, and @opentelemetry/exporter-trace-otlp-http 0.220.0. Install compatible versions together and initialize them before importing instrumented application modules.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from
  "@opentelemetry/exporter-trace-otlp-http";

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
});

sdk.start();

async function shutdown(): Promise<void> {
  await sdk.shutdown();
}

process.once("SIGTERM", () => void shutdown());
process.once("SIGINT", () => void shutdown());

Configure the exporter endpoint and headers with standard OTEL_* environment variables or your deployment secret mechanism. Do not hard-code collector credentials.

In serverless runtimes, use the platform’s supported initialization and flushing hooks. A signal handler in an ephemeral request process may never run.

Create an agent span

The current GenAI span convention names an agent invocation invoke_agent {agent name} and uses gen_ai.operation.name = "invoke_agent".

import { SpanStatusCode, trace } from "@opentelemetry/api";

const tracer = trace.getTracer("agent-runtime");

async function invokeAgent<T>(
  agentName: string,
  providerName: string,
  operation: () => Promise<T>,
): Promise<T> {
  return tracer.startActiveSpan(
    `invoke_agent ${agentName}`,
    async (span) => {
      span.setAttributes({
        "gen_ai.agent.name": agentName,
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.provider.name": providerName,
      });

      try {
        return await operation();
      } catch (error) {
        if (error instanceof Error) span.recordException(error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw error;
      } finally {
        span.end();
      }
    },
  );
}

Use a low-cardinality agent name such as researcher, not a user’s task or a dynamically generated UUID. Set the provider name consistently for the agent implementation, using a documented custom value when no well-known provider value applies. Put the workflow/run ID in an application attribute if you need correlation, and ensure that field is not sensitive.

Instrument model calls with stable attributes

For each model request, capture:

• gen_ai.operation.name, such as chat;
• gen_ai.provider.name where known;
• gen_ai.request.model, the requested model identifier;
• gen_ai.response.model, when the provider returns the actual model;
• response ID where safe and useful;
• input and output token usage;
• duration and error status; and
• streaming time to first chunk as a metric.

Do not put prompts in span names or attributes. Span attributes are often indexed, copied to several backends, and retained longer than application data.

If a provider’s automatic instrumentation already emits GenAI spans, do not create a duplicate manual span around the same network call. Add only the orchestration context the library cannot know.

Instrument tool execution

The current convention names tool spans execute_tool {tool name}. A wrapper can capture outcome and duration while leaving arguments out:

async function executeTool<T>(
  toolName: string,
  operation: () => Promise<T>,
): Promise<T> {
  return tracer.startActiveSpan(
    `execute_tool ${toolName}`,
    async (span) => {
      span.setAttributes({
        "gen_ai.operation.name": "execute_tool",
        "gen_ai.tool.name": toolName,
      });

      try {
        return await operation();
      } catch (error) {
        if (error instanceof Error) span.recordException(error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw error;
      } finally {
        span.end();
      }
    },
  );
}

Add application attributes for tool.read_or_write, tool.retry_attempt, and approval.present if those dimensions are useful and bounded. Never attach raw SQL, authorization headers, full URLs with tokens, or free-form arguments.

For an MCP tool, record the approved server identity and tool name separately. A tool name alone can collide across servers.

Represent retrieval without leaking documents

Trace retrieval as a child operation with metadata such as:

• data store and index name from an allowlisted set;
• requested and returned candidate counts;
• search mode: lexical, exact vector, approximate vector, or hybrid;
• embedding model version;
• tenant filter applied: true/false;
• latency; and
• selected memory/source IDs using opaque identifiers.

Do not export the retrieved text by default. If debugging content is essential, use an explicit incident workflow with tighter access, sampling, encryption, and deletion—not a permanent “debug=true” attribute.

Propagate context across agents and queues

In-process promise chains inherit the active OpenTelemetry context when the Node SDK is configured correctly. Across a queue or A2A boundary, inject the standard trace context into message metadata and extract it in the worker.

Treat incoming trace headers as untrusted. Accept only supported propagation fields, enforce size limits, and do not let a caller choose arbitrary baggage that your backend indexes. Baggage can contain user-controlled values and is propagated farther than span-local attributes.

For durable workflows, record both:

• a trace for each active execution segment; and
• a stable workflow/run ID that links segments across long pauses.

A trace should not remain open for days while waiting for human approval. End the active span, persist the workflow state, and link or correlate the resumed trace.

Build metrics from spans and explicit instruments

The current GenAI semantic conventions define metrics including operation duration and token usage. A useful dashboard includes:

Do not use user IDs, prompt text, trace IDs, or raw error messages as metric labels. High-cardinality labels can make telemetry expensive or unusable.

Define a redaction policy before capture

Classify fields into three groups:

1. Always allowed: agent name, approved tool name, operation, duration, token counts, result status.
2. Conditionally allowed: opaque document IDs, model response IDs, tenant pseudonyms, normalized error types.
3. Never in ordinary telemetry: credentials, cookies, prompt/output content, personal data, complete tool arguments/results, downloaded files.

Apply redaction before export, not only in the backend UI. Once a secret reaches a collector, several systems may already have copied it.

Add automated tests that send canary secrets through prompts and tool results, export to an in-memory or test collector, and assert that the canaries are absent.

Sampling and retention

Head sampling decides before the trace outcome is known, so it can miss rare failures. Tail sampling at a collector can retain errors, slow traces, and selected workflows after seeing the full trace.

A practical policy is:

• keep 100% of safety and authorization failures;
• keep all traces above a tail-latency threshold;
• keep a small random sample of successful high-volume workflows;
• increase sampling temporarily for a new release; and
• retain detailed traces for less time than aggregate metrics.

Sampling is not a privacy control. Redact first, sample second.

Use traces to improve the system

A trace is useful when it connects to a decision. Link failed CI evals to their traces. Compare the same span topology between the production baseline and candidate release. Look for:

• duplicated retrieval or tool calls;
• branches on the critical path that could run concurrently;
• retries on permanent errors;
• an agent receiving far more tokens than its task requires;
• a child span continuing after the workflow has completed; and
• tool results that trigger unrelated follow-up actions.

Those observations feed directly into bounded parallelism and cost control.

Sources and freshness notes

This guide was checked on July 18, 2026 against the current GenAI attribute registry, the separately maintained OpenTelemetry GenAI semantic-conventions repository, and the official GenAI observability guide. GenAI convention stability varies by signal; pin your convention version and review changes before renaming production attributes.