Deep Dive
12 min readOctober 9, 2026Updated Aug 2026Observability for Non-Deterministic AI: OpenTelemetry & Cost Spans
Architecting enterprise observability for non-deterministic AI agent swarms using OpenTelemetry distributed traces, loop detection, and tenant cost attribution.
Executive Summary
How enterprise engineering teams instrument OpenTelemetry spans, traces, and Prometheus metrics to observe non-deterministic autonomous agent swarms, detect recursive tool loops, and attribute token costs per tenant.
When traditional microservices fail, debugging is straightforward: inspect the HTTP status code, examine the stack trace, and reproduce the error using deterministic input payloads.
However, when **Autonomous Multi-Agent Swarms** fail in production, traditional observability tools are completely blind.
Generative AI pipelines are inherently **non-deterministic**:
- Identical user prompts can produce different reasoning chains, tool selections, and completion tokens.
- Agents can enter infinite recursive tool-calling loops without throwing unhandled exceptions.
- A single buggy customer prompt can trigger 80 consecutive LLM invocations, quietly burning $30 in cloud GPU spend before any alert fires.
text
┌────────────────────────────────────────────────────────────────────────┐
│ THE OBSERVABILITY GAP IN AGENTIC SYSTEMS │
│ │
│ Traditional APM (Datadog / New Relic): │
│ - Sees: 1 HTTP 200 OK Response (Duration: 8.4 seconds) │
│ - Blind to: 14 sub-agent tool calls, 4 hallucinations, $0.18 spend ❌ │
│ │
│ AI-Native OpenTelemetry Tracing: │
│ - Root Span: agent.execute_session (8.4s) │
│ ├── Span 1: qdrant.vector_search (42ms) │
│ ├── Span 2: gemini_flash.classify_intent (95ms | $0.0001) │
│ ├── Span 3: mcp_tool.github_api (140ms) │
│ └── Span 4: claude_sonnet.synthesize (480ms | $0.021) │
│ Result: 100% Traceability • Infinite Loop Prevention • Cost Metering │
└────────────────────────────────────────────────────────────────────────┘By instrumenting **OpenTelemetry Semantic Conventions for Generative AI**, engineering teams regain complete deterministic control over non-deterministic systems.
1. The OpenTelemetry Agent Span Hierarchy
Every multi-turn agent interaction must be structured as a nested distributed trace hierarchy:
text
┌────────────────────────────────────────────────────────────────────────┐
│ DISTRIBUTED TRACE HIERARCHY │
│ │
│ Root Trace: [Session: sess_94820] (Tenant: acme-corp) │
│ │ │
│ ├──► Span: "agent.orchestrator.plan" [Gemini 2.0 Flash] │
│ │ Attrs: { gen_ai.prompt_tokens: 820, gen_ai.cost_usd: 0.00008 } │
│ │ │
│ ├──► Span: "tool.mcp.postgres_query" [Tool Execution] │
│ │ Attrs: { db.system: "postgresql", db.statement: "SELECT ..." } │
│ │ │
│ └──► Span: "agent.evaluator.verify" [Claude 3.5 Sonnet] │
│ Attrs: { gen_ai.completion_tokens: 340, eval.passed: true } │
└────────────────────────────────────────────────────────────────────────┘2. Production OpenTelemetry Instrumentation in TypeScript
Here is the production-grade OpenTelemetry wrapper that instruments LLM requests with token counts, latencies, and tenant tags:
typescript
// src/observability/ai-tracer.ts
import { trace, SpanStatusCode, Span } from '@opentelemetry/api';
const tracer = trace.getTracer('enterprise-agent-observability', '1.0.0');
export interface AITraceContext {
tenantId: string;
sessionId: string;
modelName: string;
}
export async function traceAIOperation<T>(
spanName: string,
context: AITraceContext,
operation: (span: Span) => Promise<{ result: T; inputTokens: number; outputTokens: number }>
): Promise<T> {
return tracer.startActiveSpan(spanName, async (span) => {
const startTime = performance.now();
// Set standard OpenTelemetry GenAI semantic attributes
span.setAttribute('gen_ai.system', 'gemini');
span.setAttribute('gen_ai.request.model', context.modelName);
span.setAttribute('enterprise.tenant_id', context.tenantId);
span.setAttribute('enterprise.session_id', context.sessionId);
try {
const { result, inputTokens, outputTokens } = await operation(span);
const durationMs = performance.now() - startTime;
span.setAttribute('gen_ai.usage.prompt_tokens', inputTokens);
span.setAttribute('gen_ai.usage.completion_tokens', outputTokens);
span.setAttribute('gen_ai.duration_ms', durationMs);
// Calculate token cost in USD
const estimatedCost = (inputTokens / 1_000_000) * 0.15 + (outputTokens / 1_000_000) * 0.60;
span.setAttribute('gen_ai.usage.cost_usd', estimatedCost);
span.setStatus({ code: SpanStatusCode.OK });
return result;
} catch (error: any) {
span.setStatus({
code: SpanStatusCode.ERROR,
message: error.message,
});
span.recordException(error);
throw error;
} finally {
span.end();
}
});
}3. Detecting Infinite Agent Loops with Circuit Breaker Middleware
A non-deterministic agent that gets stuck in a tool recursion loop can exhaust API budgets in minutes.
We implement **Dynamic Recursion Depth Tracking & Circuit Breaking**:
typescript
// src/observability/loop-detector.ts
export class AgentLoopCircuitBreaker {
private callHistory: Map<string, string[]> = new Map();
private readonly maxRepeatedToolCalls = 3;
private readonly maxTotalTurns = 10;
public recordStep(sessionId: string, toolName: string, argsHash: string): boolean {
const history = this.callHistory.get(sessionId) || [];
const stepSignature = `${toolName}:${argsHash}`;
history.push(stepSignature);
this.callHistory.set(sessionId, history);
// 1. Check for infinite identical tool invocations
const identicalCalls = history.filter((sig) => sig === stepSignature).length;
if (identicalCalls >= this.maxRepeatedToolCalls) {
throw new Error(`🚨 AGENT RECURSION CIRCUIT BREAKER TRIPPED: Tool [${toolName}] invoked ${identicalCalls} times with identical arguments.`);
}
// 2. Check for overall turn explosion
if (history.length > this.maxTotalTurns) {
throw new Error(`🚨 AGENT SESSION LIMIT EXCEEDED: Session [${sessionId}] exceeded maximum ${this.maxTotalTurns} turns.`);
}
return true;
}
}4. Real-Time Prometheus Metrics & Grafana Dashboards
In addition to individual distributed spans, enterprise operators require aggregate metrics exported to **Prometheus**:
typescript
// src/observability/metrics-collector.ts
import { Counter, Histogram } from 'prom-client';
export const agentToolCallDuration = new Histogram({
name: 'agent_tool_call_duration_seconds',
help: 'Latency distribution of agent tool executions',
labelNames: ['tool_name', 'status'],
buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});
export const agentCostCounter = new Counter({
name: 'agent_cumulative_cost_dollars_total',
help: 'Total cumulative inference cost in USD',
labelNames: ['tenant_id', 'model_tier'],
});text
┌────────────────────────────────────────────────────────────────────────┐
│ GRAFANA AGENT TELEMETRY METRICS │
│ │
│ [P99 Agent Execution Latency] ──► 1.42s (Green SLA ✅) │
│ [Total 24h Spend (acme-corp)] ──► $48.20 / $500 Daily Budget Cap │
│ [Active Loop Circuit Breakers]──► 0 Tripped │
│ [Cache Hit Rate] ──► 43.8% (Upstash Redis) │
└────────────────────────────────────────────────────────────────────────┘5. Automated Testing for Telemetry & Loop Prevention
We verify telemetry span generation and circuit breaker trips using Vitest:
typescript
// tests/observability/loop-detector.test.ts
import { describe, it, expect } from 'vitest';
import { AgentLoopCircuitBreaker } from '../../src/observability/loop-detector';
describe('Agent Observability & Loop Detection Invariants', () => {
it('trips circuit breaker when an agent loops on the same tool call', () => {
const breaker = new AgentLoopCircuitBreaker();
const sessionId = 'test-session-101';
expect(breaker.recordStep(sessionId, 'searchDatabase', 'hash_abc')).toBe(true);
expect(breaker.recordStep(sessionId, 'searchDatabase', 'hash_abc')).toBe(true);
// Third identical call must trip circuit breaker
expect(() => {
breaker.recordStep(sessionId, 'searchDatabase', 'hash_abc');
}).toThrowError(/AGENT RECURSION CIRCUIT BREAKER TRIPPED/);
});
});6. Distributed Context Propagation Across Multi-Agent Sub-Graphs
When an orchestrator agent delegates sub-tasks to child agents (such as a Planner agent invoking a Coder agent and an Evaluator agent), maintaining a continuous trace context across network boundaries requires **W3C Trace Context Propagation**:
typescript
// src/observability/context-propagator.ts
import { context, propagation, trace } from '@opentelemetry/api';
export function injectTraceContextIntoAgentMessage(headers: Record<string, string>): Record<string, string> {
const activeContext = context.active();
// Inject `traceparent` and `tracestate` into outbound JSON message headers
propagation.inject(activeContext, headers);
return headers;
}
export function extractTraceContextFromAgentMessage(headers: Record<string, string>) {
// Extract parent trace context from incoming sub-agent message
const extractedContext = propagation.extract(context.active(), headers);
return extractedContext;
}text
┌────────────────────────────────────────────────────────────────────────┐
│ TRACE CONTEXT PROPAGATION MESH │
│ │
│ [Master Planner Agent] (Trace ID: `0af7651916cd43dd8448eb211c80319c`) │
│ │ │
│ ├── Injects W3C `traceparent` header into BullMQ message queue │
│ │ │
│ ▼ │
│ [Sub-Agent Worker: Coder] (Inherits Parent Trace ID) │
│ │ │
│ ▼ │
│ [Sub-Agent Worker: Evaluator] (Single Unified Honeycomb Waterfall) ✅ │
└────────────────────────────────────────────────────────────────────────┘7. Architectural Checklist for AI Telemetry Systems
To maintain enterprise observability across autonomous agent swarms:
- **Adopt OpenTelemetry GenAI Semantic Conventions:** Tag all spans with `gen_ai.request.model`, `prompt_tokens`, and `completion_tokens`.
- **Attribute Costs Per Tenant:** Record token spend directly in spans to enable real-time gross margin calculations.
- **Deploy Loop Circuit Breakers:** Automatically terminate agent sessions that repeat identical tool arguments more than three times.
- **Stream Traces to Honeycomb / Datadog via OTLP:** Use standard gRPC OpenTelemetry Collector exporters to avoid vendor lock-in.
- **Instrument MCP Tool Boundaries:** Wrap every Model Context Protocol (MCP) server call in its own dedicated span with input and output schemas.
- **Establish Real-Time Cost Ceilings:** Alert on-call engineers via Slack/PagerDuty whenever a single session consumes more than $0.50 in compute.
- **Export Aggregate Prometheus Histograms:** Track P95 and P99 latency buckets for every individual tool type across the agent fleet.
- **Propagate W3C Trace Headers:** Ensure multi-agent job queues preserve distributed parent-child span relationships across message brokers.
To learn how observability integrates into complete enterprise architectures, read our [Master Playbook on Legacy AI Modernization](/guides/legacy-system-ai-modernization-playbook) or schedule an [Observability Architecture Consultation](/services/architecture).