Telemetry

The Go AI SDK uses OpenTelemetry to collect telemetry data. OpenTelemetry is an open-source observability framework designed to provide standardized instrumentation for collecting telemetry data.

Check out the AI SDK Observability Integrations to see providers that offer monitoring and tracing for AI SDK applications.

Enabling Telemetry

Telemetry is active by default when at least one telemetry integration is registered. Use Telemetry for per-call options; ExperimentalTelemetry remains as a deprecated alias for migration.

import (
    "context"

    "github.com/digitallysavvy/go-ai/pkg/ai"
    "github.com/digitallysavvy/go-ai/pkg/telemetry"
)

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{},
})

When telemetry is enabled, you can control whether to record input values and output values. By default, both are enabled. You can disable them for privacy, data transfer, or performance reasons:

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        RecordInputs:  false, // Don't record sensitive inputs
        RecordOutputs: true,  // Record outputs
    },
})

Telemetry Settings

FunctionID

Provide a FunctionID to identify the function that the telemetry data is for:

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        FunctionID: "story-generator",
    },
})

Context Inclusion

Runtime and tool context is excluded from telemetry by default. Opt in to specific top-level keys:

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:          model,
    Prompt:         "Write a short story about a cat.",
    RuntimeContext: map[string]interface{}{"request_id": "req-123", "secret": "token"},
    ToolsContext: map[string]interface{}{
        "weather": map[string]interface{}{"city": "Paris", "api_key": "secret"},
    },
    Telemetry: &telemetry.Options{
        IncludeRuntimeContext: map[string]bool{"request_id": true},
        IncludeToolsContext: map[string]map[string]bool{
            "weather": map[string]bool{"city": true},
        },
    },
})

Custom Tracer

Provide a custom OpenTelemetry tracer instead of using the global tracer. The tracer belongs to the integration, not to telemetry.Options: construct the integration with it and register it once at startup.

import (
    "go.opentelemetry.io/otel/sdk/trace"

    "github.com/digitallysavvy/go-ai/pkg/telemetry"
)

// Create custom tracer provider
tp := trace.NewTracerProvider()
tracer := tp.Tracer("my-custom-tracer")

// GenAI semantic-convention spans. Use telemetry.NewLegacyOpenTelemetry
// with telemetry.LegacyOpenTelemetryOptions{Tracer: tracer} for the
// ai.* span names instead.
telemetry.RegisterTelemetryIntegration(
    telemetry.NewOpenTelemetry(telemetry.OpenTelemetryOptions{Tracer: tracer}),
)

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
    },
})

Custom Span Attributes

Use EnrichSpan to attach observability-specific attributes when OTel spans are created. SDK-managed attributes take precedence when a custom key overlaps with a standard AI SDK or GenAI semantic-convention key.

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a deployment summary.",
    RuntimeContext: map[string]interface{}{
        "user_id": "user-123",
    },
    Telemetry: &telemetry.Options{
        EnrichSpan: func(ctx context.Context, span telemetry.EnrichSpanOptions) map[string]interface{} {
            attrs := map[string]interface{}{
                "app.span_type": string(span.SpanType),
            }
            if span.RuntimeContext["user_id"] != nil {
                attrs["app.user_id"] = span.RuntimeContext["user_id"]
            }
            return attrs
        },
    },
})

Event Names

Telemetry integrations should implement OnEnd, OnEmbedEnd, and OnRerankEnd. The older OnFinish, OnEmbedFinish, and OnRerankFinish names are kept as deprecated compatibility fallbacks. Streaming OnChunk telemetry is no longer emitted; application-level StreamTextOptions.OnChunk callbacks still receive stream chunks.

OnEmbedEnd and OnRerankEnd now also fire for a failed model-call attempt (a retry that errored), not only on success. EmbeddingModelCallEndEvent and RerankingModelCallEndEvent each gained an Error error field: when set, the built-in integrations record error status on the span instead of setting usage/embeddings/ranking attributes. Custom integrations that implement OnEmbedEnd / OnRerankEnd should check event.Error before assuming a successful call.

Step lifecycle telemetry uses OnStepEnd as the canonical name. OnStepFinish is still emitted as a deprecated compatibility event for migration, and OnStepEnd takes precedence when both callback names are configured.

The built-in telemetry.OpenTelemetry and telemetry.LegacyOpenTelemetry integrations set the OTel span status to codes.Error on the language-model-call, step, and root spans whenever the corresponding event's FinishReason is "error" (for example a provider stream that ends with an error chunk), even when no separate OnError/OnStepError event fires. Spans finishing with any other reason keep the default Unset status.

Abort and Performance Events

Telemetry integrations can implement OnAbort(context.Context, telemetry.TelemetryAbortEvent). The SDK emits this event when generation is canceled or aborted and closes the active OpenTelemetry span so traces do not remain open.

Model calls are made inside the telemetry context returned by OnStart and OnStepStart. If an integration stores span context or request-scoped values in the returned context, provider calls and tool execution receive that context.

Language model call performance uses TypeScript-compatible names:

type LanguageModelCallPerformance struct {
    TimeToFirstOutputMs       *int64
    TimeBetweenOutputChunksMs *types.OutputChunkTimingStats
}

File outputs and tool calls count as output for TimeToFirstOutputMs. Runtime and tool context remain excluded from telemetry by default; when included through IncludeRuntimeContext or IncludeToolsContext, nested objects are emitted as separate attributes instead of a single collapsed object.

Fluent API for Settings

Use the fluent API to build telemetry settings:

telemetrySettings := telemetry.DefaultSettings().
    WithEnabled(true).
    WithFunctionID("story-generator").
    WithRecordInputs(false)

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:     model,
    Prompt:    "Generate text",
    Telemetry: telemetrySettings,
})

Setting Up OpenTelemetry in Go

Before using telemetry, you need to configure OpenTelemetry in your application:

Basic Setup

package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTelemetry() func() {
    // Create stdout exporter
    exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }

    // Create tracer provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )

    // Set global tracer provider
    otel.SetTracerProvider(tp)

    // Return cleanup function
    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}

func main() {
    // Initialize telemetry
    cleanup := initTelemetry()
    defer cleanup()

    // Your application code...
}

Jaeger Setup

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initJaeger() func() {
    // Create Jaeger exporter
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://localhost:14268/api/traces"),
    ))
    if err != nil {
        log.Fatalf("failed to create Jaeger exporter: %v", err)
    }

    // Create tracer provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithSampler(trace.AlwaysSample()),
    )

    otel.SetTracerProvider(tp)

    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}

OTLP (OpenTelemetry Protocol) Setup

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initOTLP() func() {
    // Create OTLP exporter
    exporter, err := otlptracehttp.New(context.Background(),
        otlptracehttp.WithEndpoint("localhost:4318"),
        otlptracehttp.WithInsecure(),
    )
    if err != nil {
        log.Fatalf("failed to create OTLP exporter: %v", err)
    }

    // Create tracer provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )

    otel.SetTracerProvider(tp)

    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}

Collected Data

The Go AI SDK records spans and attributes following OpenTelemetry conventions.

GenerateText

Records the following spans:

ai.generateText span:

• Full length of the generateText call
• Contains one or more ai.generateText.doGenerate spans
• Attributes:
  • operation.name: ai.generateText + function ID
  • ai.operationId: "ai.generateText"
  • ai.model.id: Model identifier
  • ai.model.provider: Provider name
  • ai.prompt: The prompt used
  • ai.response.text: Generated text (if RecordOutputs is true)
  • ai.response.finishReason: Finish reason
  • ai.usage.promptTokens: Input tokens used
  • ai.usage.completionTokens: Output tokens used
  • ai.usage.totalTokens: Total tokens used
  • ai.telemetry.functionId: Function ID from settings
  • Custom metadata attributes

ai.generateText.doGenerate span:

• Individual provider call
• Attributes:
  • operation.name: ai.generateText.doGenerate + function ID
  • ai.operationId: "ai.generateText.doGenerate"
  • ai.prompt.messages: Messages sent to provider
  • ai.response.text: Generated text
  • ai.response.finishReason: Finish reason
  • ai.response.model: Actual model used by provider
  • gen_ai.system: Provider name
  • gen_ai.request.model: Requested model
  • gen_ai.request.temperature: Temperature setting
  • gen_ai.request.max_tokens: Max tokens setting
  • gen_ai.usage.input_tokens: Input tokens
  • gen_ai.usage.output_tokens: Output tokens

StreamText

Records the following spans:

ai.streamText span:

• Full length of the streamText call
• Contains ai.streamText.doStream span
• Same attributes as ai.generateText, plus:
  • ai.response.msToFirstChunk: Time to first chunk (milliseconds)
  • ai.response.msToFinish: Time to completion (milliseconds)
  • ai.response.effectiveOutputTokensPerSecond: Output tokens per request second
  • ai.response.effectiveTotalTokensPerSecond: Input plus output tokens per request second
  • ai.response.outputTokensPerSecond: Streaming output tokens per output-stream second, when available
  • ai.response.inputTokensPerSecond: Streaming input tokens per time-to-first-output-token second, when available

ai.streamText.doStream span:

• Individual provider stream call
• Emits ai.stream.firstChunk event when first chunk received
• Same attributes as ai.generateText.doGenerate, plus streaming metrics

GenerateObject

Records the following spans:

ai.generateObject span:

• Full length of the generateObject call
• Attributes:
  • Same as ai.generateText, plus:
  • ai.schema: Stringified JSON schema
  • ai.schema.name: Schema name
  • ai.schema.description: Schema description
  • ai.response.object: Generated object (stringified JSON)
  • ai.settings.output: Output type (object, array, no-schema, etc.)

ai.generateObject.doGenerate span:

• Individual provider call
• Attributes similar to ai.generateText.doGenerate

StreamObject

Records the following spans:

ai.streamObject span:

• Full length of the streamObject call
• Same attributes as ai.generateObject, plus streaming metrics

ai.streamObject.doStream span:

• Individual provider stream call
• Emits ai.stream.firstChunk event

Embed

Records the following spans:

ai.embed span:

• Full length of the embed call
• Attributes:
  • operation.name: ai.embed + function ID
  • ai.operationId: "ai.embed"
  • ai.model.id: Model identifier
  • ai.model.provider: Provider name
  • ai.value: Input value (if RecordInputs is true)
  • ai.embedding: Stringified embedding vector (if RecordOutputs is true)
  • ai.usage.tokens: Tokens used

ai.embed.doEmbed span:

• Provider embed call
• Similar attributes

EmbedMany

Records the following spans:

ai.embedMany span:

• Full length of the embedMany call
• Attributes:
  • operation.name: ai.embedMany + function ID
  • ai.operationId: "ai.embedMany"
  • ai.values: Input values array
  • ai.embeddings: Array of stringified embeddings
  • ai.usage.tokens: Total tokens used

ai.embedMany.doEmbed span:

• Individual provider call
• May be multiple spans if batching occurs

Speech and Transcription

GenerateSpeech, Transcribe, and ExperimentalStreamTranscribe emit a single root span covering the whole call (including retries for the non-streaming calls) — there is no nested per-attempt span, unlike ai.embed/ai.generateText. All three accept a Telemetry/ExperimentalTelemetry option, dispatch OnStart/OnEnd/OnError to every registered integration, and publish the same diagnostics-channel events (onStart, onEnd, onError) as every other operation.

ai.generateSpeech span:

• Attributes:
  • operation.name: ai.generateSpeech + function ID
  • ai.operationId: "ai.generateSpeech"
  • gen_ai.output.type: "speech" (GenAI integration only)
  • ai.request.text: Input text (if RecordInputs is true)
  • ai.response.audio.size / ai.response.audio.mediaType / ai.response.audio.format: Generated audio metadata (if RecordOutputs is true)
  • ai.usage.<key> / gen_ai.usage.<key>: Provider-reported usage, flattened to numeric attributes (e.g. ai.usage.characters for Deepgram TTS; see "Provider usage attributes" below)
  • ai.response.usage: Raw provider usage object, JSON-encoded
  • ai.response.providerMetadata: Provider-specific metadata (if enabled)
• Ends with codes.Error status on failure, including a provider response with no audio.

ai.transcribe / ai.streamTranscribe spans:

• Attributes:
  • operation.name: ai.transcribe/ai.streamTranscribe + function ID
  • ai.operationId: "ai.transcribe" / "ai.streamTranscribe"
  • gen_ai.request.stream: true for ai.streamTranscribe, false for ai.transcribe (GenAI integration only)
  • ai.request.audio.size / ai.request.audio.mediaType: Input audio metadata (if RecordInputs is true). For ai.streamTranscribe, the byte count accumulates as audio chunks are read and is only set once the stream reaches its final part.
  • ai.response.text: Transcript text (if RecordOutputs is true)
  • ai.usage.<key> / gen_ai.usage.<key>: Provider-reported usage, flattened to numeric attributes
  • ai.response.usage: Raw provider usage object, JSON-encoded
  • ai.response.providerMetadata: Provider-specific metadata (if enabled)
• Ends with codes.Error status on failure, including a provider response with no transcript.

Provider usage attributes: Speech/transcription providers report usage in their own native shape (e.g. Deepgram TTS: {"characters": N}; Deepgram transcription: {"seconds": N}; Google: {"promptTokenCount": N, ...}). The telemetry layer flattens this generically: common field spellings (inputTokens, promptTokens, promptTokenCount, totalInputTokens → input_tokens; outputTokens, completionTokens, completionTokenCount, candidatesTokenCount → output_tokens; totalTokens, totalTokenCount → total_tokens) are aliased to a shared attribute name, and everything else is converted from camelCase to snake_case (e.g. characters stays characters, promptAudioSeconds becomes prompt_audio_seconds). Nested usage objects are flattened with dotted paths.

Tool Calls

When tools are executed, ai.toolCall spans are recorded:

• operation.name: "ai.toolCall"
• ai.operationId: "ai.toolCall"
• ai.toolCall.name: Tool name
• ai.toolCall.id: Tool call ID
• ai.toolCall.args: Tool input arguments
• ai.toolCall.result: Tool output (if successful and serializable)

Example: Complete Telemetry Setup

Here's a complete example showing telemetry setup and usage:

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/digitallysavvy/go-ai/pkg/ai"
    "github.com/digitallysavvy/go-ai/pkg/providers/openai"
    "github.com/digitallysavvy/go-ai/pkg/telemetry"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTelemetry() func() {
    exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithSampler(trace.AlwaysSample()),
    )

    otel.SetTracerProvider(tp)

    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}

func main() {
    // Initialize telemetry
    cleanup := initTelemetry()
    defer cleanup()

    ctx := context.Background()

    // Set up provider and model
    provider := openai.New(openai.Config{
        APIKey: os.Getenv("OPENAI_API_KEY"),
    })
    model, _ := provider.LanguageModel("gpt-4")

    // Configure telemetry settings
    telemetrySettings := &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        RecordInputs:  true,
        RecordOutputs: true,
        FunctionID:    "story-generator",
    }

    // Generate text with telemetry
    result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
        Model:     model,
        Prompt:    "Write a short story about a cat.",
        Telemetry: telemetrySettings,
    })
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(result.Text)
}

Best Practices

1. Disable Sensitive Data Recording

Disable input/output recording for sensitive data:

telemetrySettings := &telemetry.Options{
    IsEnabled: telemetry.Bool(true),
    RecordInputs:  false, // Don't record sensitive prompts
    RecordOutputs: false, // Don't record sensitive responses
    FunctionID:    "sensitive-operation",
}

2. Use Meaningful Function IDs

Use descriptive function IDs to group related operations:

// Good
FunctionID: "user-chat-response"
FunctionID: "document-summarization"
FunctionID: "code-generation"

// Bad
FunctionID: "func1"
FunctionID: "test"

3. Include Safe Context Explicitly

Include only non-sensitive context keys that help with debugging and analysis:

Telemetry: &telemetry.Options{
    IncludeRuntimeContext: map[string]bool{
        "request_id": true,
        "feature_flag": true,
    },
}

4. Use Sampling for High-Volume Applications

For high-traffic applications, use sampling to reduce overhead:

import "go.opentelemetry.io/otel/sdk/trace"

tp := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
    trace.WithSampler(trace.TraceIDRatioBased(0.1)), // Sample 10% of traces
)

5. Configure Appropriate Exporters

Choose exporters based on your observability stack:

• Stdout: Development and debugging
• Jaeger: Self-hosted tracing
• OTLP: Cloud providers (Datadog, New Relic, Honeycomb, etc.)
• Zipkin: Existing Zipkin infrastructure

6. Monitor Performance Impact

Telemetry has a small performance cost. Monitor and adjust:

// Disable telemetry for performance-critical paths
criticalOp, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:     model,
    Prompt:    prompt,
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(false),
    },
})

// Enable telemetry for monitored paths
monitoredOp, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:     model,
    Prompt:    prompt,
    Telemetry: telemetrySettings,
})

7. Use Environment-Based Configuration

Configure telemetry based on environment:

func getTelemetrySettings(env string) *telemetry.Options {
    if env == "production" {
        return &telemetry.Options{
            IsEnabled: telemetry.Bool(true),
            RecordInputs:  false, // Privacy in production
            RecordOutputs: false,
            FunctionID:    "prod-operation",
        }
    }

    // Full telemetry in development
    return &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        RecordInputs:  true,
        RecordOutputs: true,
        FunctionID:    "dev-operation",
    }
}

OpenTelemetry Semantic Conventions

The Go AI SDK follows OpenTelemetry Semantic Conventions for GenAI operations, including:

• gen_ai.system: Provider identifier
• gen_ai.request.model: Requested model
• gen_ai.request.temperature: Temperature setting
• gen_ai.request.max_tokens: Maximum tokens
• gen_ai.response.finish_reasons: Finish reasons
• gen_ai.usage.input_tokens: Input token count
• gen_ai.usage.output_tokens: Output token count

See Also

• Error Handling
• Testing
• Middleware
• OpenTelemetry Go Documentation