# Telemetry

The Go AI SDK uses [OpenTelemetry](https://opentelemetry.io/) to collect telemetry data. OpenTelemetry is an open-source observability framework designed to provide standardized instrumentation for collecting telemetry data.

Check out the AI SDK Observability Integrations to see providers that offer monitoring and tracing for AI SDK applications.

## Enabling Telemetry

Telemetry is active by default when at least one telemetry integration is registered. Use `Telemetry` for per-call options; `ExperimentalTelemetry` remains as a deprecated alias for migration.

```go
import (
    "context"

    "github.com/digitallysavvy/go-ai/pkg/ai"
    "github.com/digitallysavvy/go-ai/pkg/telemetry"
)

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{},
})
```

When telemetry is enabled, you can control whether to record input values and output values. By default, both are enabled. You can disable them for privacy, data transfer, or performance reasons:

```go
result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        RecordInputs:  false, // Don't record sensitive inputs
        RecordOutputs: true,  // Record outputs
    },
})
```

## Telemetry Settings

### FunctionID

Provide a `FunctionID` to identify the function that the telemetry data is for:

```go
result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        FunctionID: "story-generator",
    },
})
```

### Context Inclusion

Runtime and tool context is excluded from telemetry by default. Opt in to specific top-level keys:

```go
result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:          model,
    Prompt:         "Write a short story about a cat.",
    RuntimeContext: map[string]interface{}{"request_id": "req-123", "secret": "token"},
    ToolsContext: map[string]interface{}{
        "weather": map[string]interface{}{"city": "Paris", "api_key": "secret"},
    },
    Telemetry: &telemetry.Options{
        IncludeRuntimeContext: map[string]bool{"request_id": true},
        IncludeToolsContext: map[string]map[string]bool{
            "weather": map[string]bool{"city": true},
        },
    },
})
```

### Custom Tracer

Provide a custom OpenTelemetry tracer instead of using the global tracer. The tracer belongs to the integration, not to `telemetry.Options`: construct the integration with it and register it once at startup.

```go
import (
    "go.opentelemetry.io/otel/sdk/trace"

    "github.com/digitallysavvy/go-ai/pkg/telemetry"
)

// Create custom tracer provider
tp := trace.NewTracerProvider()
tracer := tp.Tracer("my-custom-tracer")

// GenAI semantic-convention spans. Use telemetry.NewLegacyOpenTelemetry
// with telemetry.LegacyOpenTelemetryOptions{Tracer: tracer} for the
// ai.* span names instead.
telemetry.RegisterTelemetryIntegration(
    telemetry.NewOpenTelemetry(telemetry.OpenTelemetryOptions{Tracer: tracer}),
)

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a short story about a cat.",
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
    },
})
```

### Custom Span Attributes

Use `EnrichSpan` to attach observability-specific attributes when OTel spans are created. SDK-managed attributes take precedence when a custom key overlaps with a standard AI SDK or GenAI semantic-convention key.

```go
result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Write a deployment summary.",
    RuntimeContext: map[string]interface{}{
        "user_id": "user-123",
    },
    Telemetry: &telemetry.Options{
        EnrichSpan: func(ctx context.Context, span telemetry.EnrichSpanOptions) map[string]interface{} {
            attrs := map[string]interface{}{
                "app.span_type": string(span.SpanType),
            }
            if span.RuntimeContext["user_id"] != nil {
                attrs["app.user_id"] = span.RuntimeContext["user_id"]
            }
            return attrs
        },
    },
})
```

### Event Names

Telemetry integrations should implement `OnEnd`, `OnEmbedEnd`, and `OnRerankEnd`. The older `OnFinish`, `OnEmbedFinish`, and `OnRerankFinish` names are kept as deprecated compatibility fallbacks. Streaming `OnChunk` telemetry is no longer emitted; application-level `StreamTextOptions.OnChunk` callbacks still receive stream chunks.

`OnEmbedEnd` and `OnRerankEnd` now also fire for a failed model-call attempt
(a retry that errored), not only on success. `EmbeddingModelCallEndEvent` and
`RerankingModelCallEndEvent` each gained an `Error error` field: when set,
the built-in integrations record error status on the span instead of setting
usage/embeddings/ranking attributes. Custom integrations that implement
`OnEmbedEnd` / `OnRerankEnd` should check `event.Error` before assuming a
successful call.

Step lifecycle telemetry uses `OnStepEnd` as the canonical name. `OnStepFinish` is still emitted as a deprecated compatibility event for migration, and `OnStepEnd` takes precedence when both callback names are configured.

The built-in `telemetry.OpenTelemetry` and `telemetry.LegacyOpenTelemetry` integrations set the OTel span status to `codes.Error` on the language-model-call, step, and root spans whenever the corresponding event's `FinishReason` is `"error"` (for example a provider stream that ends with an error chunk), even when no separate `OnError`/`OnStepError` event fires. Spans finishing with any other reason keep the default `Unset` status.

### Abort and Performance Events

Telemetry integrations can implement `OnAbort(context.Context, telemetry.TelemetryAbortEvent)`. The SDK emits this event when generation is canceled or aborted and closes the active OpenTelemetry span so traces do not remain open.

Model calls are made inside the telemetry context returned by `OnStart` and `OnStepStart`. If an integration stores span context or request-scoped values in the returned context, provider calls and tool execution receive that context.

Language model call performance uses TypeScript-compatible names:

```go
type LanguageModelCallPerformance struct {
    TimeToFirstOutputMs       *int64
    TimeBetweenOutputChunksMs *types.OutputChunkTimingStats
}
```

File outputs and tool calls count as output for `TimeToFirstOutputMs`. Runtime and tool context remain excluded from telemetry by default; when included through `IncludeRuntimeContext` or `IncludeToolsContext`, nested objects are emitted as separate attributes instead of a single collapsed object.

## Fluent API for Settings

Use the fluent API to build telemetry settings:

```go
telemetrySettings := telemetry.DefaultSettings().
    WithEnabled(true).
    WithFunctionID("story-generator").
    WithRecordInputs(false)

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:     model,
    Prompt:    "Generate text",
    Telemetry: telemetrySettings,
})
```

## Setting Up OpenTelemetry in Go

Before using telemetry, you need to configure OpenTelemetry in your application:

### Basic Setup

```go
package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTelemetry() func() {
    // Create stdout exporter
    exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }

    // Create tracer provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )

    // Set global tracer provider
    otel.SetTracerProvider(tp)

    // Return cleanup function
    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}

func main() {
    // Initialize telemetry
    cleanup := initTelemetry()
    defer cleanup()

    // Your application code...
}
```

### Jaeger Setup

```go
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initJaeger() func() {
    // Create Jaeger exporter
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://localhost:14268/api/traces"),
    ))
    if err != nil {
        log.Fatalf("failed to create Jaeger exporter: %v", err)
    }

    // Create tracer provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithSampler(trace.AlwaysSample()),
    )

    otel.SetTracerProvider(tp)

    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}
```

### OTLP (OpenTelemetry Protocol) Setup

```go
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initOTLP() func() {
    // Create OTLP exporter
    exporter, err := otlptracehttp.New(context.Background(),
        otlptracehttp.WithEndpoint("localhost:4318"),
        otlptracehttp.WithInsecure(),
    )
    if err != nil {
        log.Fatalf("failed to create OTLP exporter: %v", err)
    }

    // Create tracer provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )

    otel.SetTracerProvider(tp)

    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}
```

## Collected Data

The Go AI SDK records spans and attributes following OpenTelemetry conventions.

### GenerateText

Records the following spans:

**`ai.generateText` span:**
- Full length of the generateText call
- Contains one or more `ai.generateText.doGenerate` spans
- Attributes:
  - `operation.name`: `ai.generateText` + function ID
  - `ai.operationId`: `"ai.generateText"`
  - `ai.model.id`: Model identifier
  - `ai.model.provider`: Provider name
  - `ai.prompt`: The prompt used
  - `ai.response.text`: Generated text (if RecordOutputs is true)
  - `ai.response.finishReason`: Finish reason
  - `ai.usage.promptTokens`: Input tokens used
  - `ai.usage.completionTokens`: Output tokens used
  - `ai.usage.totalTokens`: Total tokens used
  - `ai.telemetry.functionId`: Function ID from settings
  - Custom metadata attributes

**`ai.generateText.doGenerate` span:**
- Individual provider call
- Attributes:
  - `operation.name`: `ai.generateText.doGenerate` + function ID
  - `ai.operationId`: `"ai.generateText.doGenerate"`
  - `ai.prompt.messages`: Messages sent to provider
  - `ai.response.text`: Generated text
  - `ai.response.finishReason`: Finish reason
  - `ai.response.model`: Actual model used by provider
  - `gen_ai.system`: Provider name
  - `gen_ai.request.model`: Requested model
  - `gen_ai.request.temperature`: Temperature setting
  - `gen_ai.request.max_tokens`: Max tokens setting
  - `gen_ai.usage.input_tokens`: Input tokens
  - `gen_ai.usage.output_tokens`: Output tokens

### StreamText

Records the following spans:

**`ai.streamText` span:**
- Full length of the streamText call
- Contains `ai.streamText.doStream` span
- Same attributes as `ai.generateText`, plus:
  - `ai.response.msToFirstChunk`: Time to first chunk (milliseconds)
  - `ai.response.msToFinish`: Time to completion (milliseconds)
  - `ai.response.effectiveOutputTokensPerSecond`: Output tokens per request second
  - `ai.response.effectiveTotalTokensPerSecond`: Input plus output tokens per request second
  - `ai.response.outputTokensPerSecond`: Streaming output tokens per output-stream second, when available
  - `ai.response.inputTokensPerSecond`: Streaming input tokens per time-to-first-output-token second, when available

**`ai.streamText.doStream` span:**
- Individual provider stream call
- Emits `ai.stream.firstChunk` event when first chunk received
- Same attributes as `ai.generateText.doGenerate`, plus streaming metrics

### GenerateObject

Records the following spans:

**`ai.generateObject` span:**
- Full length of the generateObject call
- Attributes:
  - Same as `ai.generateText`, plus:
  - `ai.schema`: Stringified JSON schema
  - `ai.schema.name`: Schema name
  - `ai.schema.description`: Schema description
  - `ai.response.object`: Generated object (stringified JSON)
  - `ai.settings.output`: Output type (`object`, `array`, `no-schema`, etc.)

**`ai.generateObject.doGenerate` span:**
- Individual provider call
- Attributes similar to `ai.generateText.doGenerate`

### StreamObject

Records the following spans:

**`ai.streamObject` span:**
- Full length of the streamObject call
- Same attributes as `ai.generateObject`, plus streaming metrics

**`ai.streamObject.doStream` span:**
- Individual provider stream call
- Emits `ai.stream.firstChunk` event

### Embed

Records the following spans:

**`ai.embed` span:**
- Full length of the embed call
- Attributes:
  - `operation.name`: `ai.embed` + function ID
  - `ai.operationId`: `"ai.embed"`
  - `ai.model.id`: Model identifier
  - `ai.model.provider`: Provider name
  - `ai.value`: Input value (if RecordInputs is true)
  - `ai.embedding`: Stringified embedding vector (if RecordOutputs is true)
  - `ai.usage.tokens`: Tokens used

**`ai.embed.doEmbed` span:**
- Provider embed call
- Similar attributes

### EmbedMany

Records the following spans:

**`ai.embedMany` span:**
- Full length of the embedMany call
- Attributes:
  - `operation.name`: `ai.embedMany` + function ID
  - `ai.operationId`: `"ai.embedMany"`
  - `ai.values`: Input values array
  - `ai.embeddings`: Array of stringified embeddings
  - `ai.usage.tokens`: Total tokens used

**`ai.embedMany.doEmbed` span:**
- Individual provider call
- May be multiple spans if batching occurs

### Speech and Transcription

`GenerateSpeech`, `Transcribe`, and `ExperimentalStreamTranscribe` emit a single root span covering the whole call (including retries for the non-streaming calls) — there is no nested per-attempt span, unlike `ai.embed`/`ai.generateText`. All three accept a `Telemetry`/`ExperimentalTelemetry` option, dispatch `OnStart`/`OnEnd`/`OnError` to every registered integration, and publish the same diagnostics-channel events (`onStart`, `onEnd`, `onError`) as every other operation.

**`ai.generateSpeech` span:**
- Attributes:
  - `operation.name`: `ai.generateSpeech` + function ID
  - `ai.operationId`: `"ai.generateSpeech"`
  - `gen_ai.output.type`: `"speech"` (GenAI integration only)
  - `ai.request.text`: Input text (if RecordInputs is true)
  - `ai.response.audio.size` / `ai.response.audio.mediaType` / `ai.response.audio.format`: Generated audio metadata (if RecordOutputs is true)
  - `ai.usage.<key>` / `gen_ai.usage.<key>`: Provider-reported usage, flattened to numeric attributes (e.g. `ai.usage.characters` for Deepgram TTS; see "Provider usage attributes" below)
  - `ai.response.usage`: Raw provider usage object, JSON-encoded
  - `ai.response.providerMetadata`: Provider-specific metadata (if enabled)
- Ends with `codes.Error` status on failure, including a provider response with no audio.

**`ai.transcribe` / `ai.streamTranscribe` spans:**
- Attributes:
  - `operation.name`: `ai.transcribe`/`ai.streamTranscribe` + function ID
  - `ai.operationId`: `"ai.transcribe"` / `"ai.streamTranscribe"`
  - `gen_ai.request.stream`: `true` for `ai.streamTranscribe`, `false` for `ai.transcribe` (GenAI integration only)
  - `ai.request.audio.size` / `ai.request.audio.mediaType`: Input audio metadata (if RecordInputs is true). For `ai.streamTranscribe`, the byte count accumulates as audio chunks are read and is only set once the stream reaches its final part.
  - `ai.response.text`: Transcript text (if RecordOutputs is true)
  - `ai.usage.<key>` / `gen_ai.usage.<key>`: Provider-reported usage, flattened to numeric attributes
  - `ai.response.usage`: Raw provider usage object, JSON-encoded
  - `ai.response.providerMetadata`: Provider-specific metadata (if enabled)
- Ends with `codes.Error` status on failure, including a provider response with no transcript.

**Provider usage attributes:** Speech/transcription providers report usage in their own native shape (e.g. Deepgram TTS: `{"characters": N}`; Deepgram transcription: `{"seconds": N}`; Google: `{"promptTokenCount": N, ...}`). The telemetry layer flattens this generically: common field spellings (`inputTokens`, `promptTokens`, `promptTokenCount`, `totalInputTokens` → `input_tokens`; `outputTokens`, `completionTokens`, `completionTokenCount`, `candidatesTokenCount` → `output_tokens`; `totalTokens`, `totalTokenCount` → `total_tokens`) are aliased to a shared attribute name, and everything else is converted from camelCase to `snake_case` (e.g. `characters` stays `characters`, `promptAudioSeconds` becomes `prompt_audio_seconds`). Nested usage objects are flattened with dotted paths.

### Tool Calls

When tools are executed, `ai.toolCall` spans are recorded:

- `operation.name`: `"ai.toolCall"`
- `ai.operationId`: `"ai.toolCall"`
- `ai.toolCall.name`: Tool name
- `ai.toolCall.id`: Tool call ID
- `ai.toolCall.args`: Tool input arguments
- `ai.toolCall.result`: Tool output (if successful and serializable)

## Example: Complete Telemetry Setup

Here's a complete example showing telemetry setup and usage:

```go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/digitallysavvy/go-ai/pkg/ai"
    "github.com/digitallysavvy/go-ai/pkg/providers/openai"
    "github.com/digitallysavvy/go-ai/pkg/telemetry"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTelemetry() func() {
    exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithSampler(trace.AlwaysSample()),
    )

    otel.SetTracerProvider(tp)

    return func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }
}

func main() {
    // Initialize telemetry
    cleanup := initTelemetry()
    defer cleanup()

    ctx := context.Background()

    // Set up provider and model
    provider := openai.New(openai.Config{
        APIKey: os.Getenv("OPENAI_API_KEY"),
    })
    model, _ := provider.LanguageModel("gpt-4")

    // Configure telemetry settings
    telemetrySettings := &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        RecordInputs:  true,
        RecordOutputs: true,
        FunctionID:    "story-generator",
    }

    // Generate text with telemetry
    result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
        Model:     model,
        Prompt:    "Write a short story about a cat.",
        Telemetry: telemetrySettings,
    })
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(result.Text)
}
```

## Best Practices

### 1. Disable Sensitive Data Recording

Disable input/output recording for sensitive data:

```go
telemetrySettings := &telemetry.Options{
    IsEnabled: telemetry.Bool(true),
    RecordInputs:  false, // Don't record sensitive prompts
    RecordOutputs: false, // Don't record sensitive responses
    FunctionID:    "sensitive-operation",
}
```

### 2. Use Meaningful Function IDs

Use descriptive function IDs to group related operations:

```go
// Good
FunctionID: "user-chat-response"
FunctionID: "document-summarization"
FunctionID: "code-generation"

// Bad
FunctionID: "func1"
FunctionID: "test"
```

### 3. Include Safe Context Explicitly

Include only non-sensitive context keys that help with debugging and analysis:

```go
Telemetry: &telemetry.Options{
    IncludeRuntimeContext: map[string]bool{
        "request_id": true,
        "feature_flag": true,
    },
}
```

### 4. Use Sampling for High-Volume Applications

For high-traffic applications, use sampling to reduce overhead:

```go
import "go.opentelemetry.io/otel/sdk/trace"

tp := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
    trace.WithSampler(trace.TraceIDRatioBased(0.1)), // Sample 10% of traces
)
```

### 5. Configure Appropriate Exporters

Choose exporters based on your observability stack:

- **Stdout**: Development and debugging
- **Jaeger**: Self-hosted tracing
- **OTLP**: Cloud providers (Datadog, New Relic, Honeycomb, etc.)
- **Zipkin**: Existing Zipkin infrastructure

### 6. Monitor Performance Impact

Telemetry has a small performance cost. Monitor and adjust:

```go
// Disable telemetry for performance-critical paths
criticalOp, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:     model,
    Prompt:    prompt,
    Telemetry: &telemetry.Options{
        IsEnabled: telemetry.Bool(false),
    },
})

// Enable telemetry for monitored paths
monitoredOp, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:     model,
    Prompt:    prompt,
    Telemetry: telemetrySettings,
})
```

### 7. Use Environment-Based Configuration

Configure telemetry based on environment:

```go
func getTelemetrySettings(env string) *telemetry.Options {
    if env == "production" {
        return &telemetry.Options{
            IsEnabled: telemetry.Bool(true),
            RecordInputs:  false, // Privacy in production
            RecordOutputs: false,
            FunctionID:    "prod-operation",
        }
    }

    // Full telemetry in development
    return &telemetry.Options{
        IsEnabled: telemetry.Bool(true),
        RecordInputs:  true,
        RecordOutputs: true,
        FunctionID:    "dev-operation",
    }
}
```

## OpenTelemetry Semantic Conventions

The Go AI SDK follows [OpenTelemetry Semantic Conventions for GenAI operations](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/), including:

- `gen_ai.system`: Provider identifier
- `gen_ai.request.model`: Requested model
- `gen_ai.request.temperature`: Temperature setting
- `gen_ai.request.max_tokens`: Maximum tokens
- `gen_ai.response.finish_reasons`: Finish reasons
- `gen_ai.usage.input_tokens`: Input token count
- `gen_ai.usage.output_tokens`: Output token count

## See Also

- [Error Handling](https://goaisdk.com/docs/ai-sdk-core/error-handling.md)
- [Testing](https://goaisdk.com/docs/ai-sdk-core/testing.md)
- [Middleware](https://goaisdk.com/docs/ai-sdk-core/middleware.md)
- [OpenTelemetry Go Documentation](https://opentelemetry.io/docs/instrumentation/go/)