Memory Optimization

Learn how to reduce memory consumption by 50-80% using retention settings.

Overview

The Go AI SDK includes retention settings that control what data is retained from LLM requests and responses. By excluding request and response bodies from results, you can significantly reduce memory usage while maintaining all essential metadata.

When to Use Retention Settings

Image Processing Workloads

When sending images in prompts (vision models), the base64-encoded images can be 1-5 MB each. By excluding the request body, you avoid storing these large images in memory:

retention := &types.RetentionSettings{
    RequestBody: types.BoolPtr(false), // Don't store the image
}

result, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: imagePrompt, // Contains large image
    ExperimentalRetention: retention,
})

Memory savings: 50-80% per request

Large Context Windows

When analyzing documents, code repositories, or long conversations, the context can be 10-100 KB per request:

retention := &types.RetentionSettings{
    RequestBody:  types.BoolPtr(false), // Don't store large context
    ResponseBody: types.BoolPtr(false), // Don't store large responses
}

result, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:                 model,
    Messages:              longConversation, // Large message history
    ExperimentalRetention: retention,
})

Memory savings: 40-60% per request

Long-Running Services

For services that process thousands of requests, memory accumulation can lead to:

• High memory usage
• Frequent garbage collection
• Potential out-of-memory errors

Retention settings prevent memory accumulation over time.

Privacy-Sensitive Applications

Exclude sensitive data from being stored:

• Confidential user inputs
• Personal information
• Proprietary content

retention := &types.RetentionSettings{
    RequestBody:  types.BoolPtr(false), // Don't store user data
    ResponseBody: types.BoolPtr(false), // Don't store AI responses
}

Usage

Basic Usage

import (
    "context"
    "github.com/digitallysavvy/go-ai/pkg/ai"
    "github.com/digitallysavvy/go-ai/pkg/provider/types"
)

// Create retention settings
retention := &types.RetentionSettings{
    RequestBody:  types.BoolPtr(false),
    ResponseBody: types.BoolPtr(false),
}

// Use with GenerateText
result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:                 model,
    Prompt:                prompt,
    ExperimentalRetention: retention,
})

Retention Options

The RetentionSettings struct has two fields:

type RetentionSettings struct {
    // RequestBody controls whether to retain the request body
    // nil = default (retain), false = exclude, true = explicitly retain
    RequestBody *bool

    // ResponseBody controls whether to retain the response body
    // nil = default (retain), false = exclude, true = explicitly retain
    ResponseBody *bool
}

Helper function:

types.BoolPtr(false) // Creates *bool pointing to false

Default Behavior

By default (when ExperimentalRetention is nil), all data is retained for backwards compatibility:

result, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: prompt,
    // ExperimentalRetention is nil - retains everything
})

// result.RawRequest and result.RawResponse are present

Exclude Request Only

Useful when you want to save the response for debugging but not the input:

retention := &types.RetentionSettings{
    RequestBody: types.BoolPtr(false), // Exclude request
    // ResponseBody is nil, so response is retained
}

Exclude Response Only

Useful when you want to audit inputs but not outputs:

retention := &types.RetentionSettings{
    // RequestBody is nil, so request is retained
    ResponseBody: types.BoolPtr(false), // Exclude response
}

Exclude Both

Maximum memory savings:

retention := &types.RetentionSettings{
    RequestBody:  types.BoolPtr(false),
    ResponseBody: types.BoolPtr(false),
}

What's Still Available

Even with retention disabled, you still get:

• ✅ Generated text content
• ✅ Token usage information (Usage)
• ✅ Finish reason (FinishReason)
• ✅ Tool calls and results
• ✅ Warnings
• ✅ All metadata

Only the raw request/response bodies (RawRequest, RawResponse) are excluded.

Streaming Support

Retention settings work with StreamText as well:

stream, _ := ai.StreamText(ctx, ai.StreamTextOptions{
    Model:                 model,
    Prompt:                prompt,
    ExperimentalRetention: retention,
})

Performance Impact

Memory Savings

Typical memory reduction for different workloads:

Workload Type	Without Retention	With Retention	Savings
Text-only	0.1-0.5 MB	0.05-0.1 MB	40-50%
Single image	2-5 MB	0.1-0.5 MB	80-90%
Multiple images	10-20 MB	0.2-0.8 MB	90-95%
Large documents	1-3 MB	0.1-0.3 MB	70-80%
1000 requests (avg)	500 MB	100 MB	80%

No Performance Overhead

Retention settings have zero performance overhead:

• No extra processing when enabled
• No slowdown in generation
• Only affects memory storage

Garbage Collection Benefits

Lower memory usage leads to:

• Fewer GC cycles
• Shorter GC pause times
• Better overall application performance

Best Practices

1. Enable in Production

For production applications, enable retention by default:

// config.go
var DefaultRetention = &types.RetentionSettings{
    RequestBody:  types.BoolPtr(false),
    ResponseBody: types.BoolPtr(false),
}

// usage
result, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:                 model,
    Prompt:                prompt,
    ExperimentalRetention: config.DefaultRetention,
})

2. Disable in Development

During development, retain data for debugging:

func getRetentionSettings() *types.RetentionSettings {
    if os.Getenv("ENV") == "production" {
        return &types.RetentionSettings{
            RequestBody:  types.BoolPtr(false),
            ResponseBody: types.BoolPtr(false),
        }
    }
    return nil // Retain everything in development
}

3. Conditional Retention

Apply retention based on prompt characteristics:

func getRetentionForPrompt(hasImages bool) *types.RetentionSettings {
    if hasImages {
        // Images are large, exclude request
        return &types.RetentionSettings{
            RequestBody: types.BoolPtr(false),
        }
    }
    return nil // Text-only prompts, no retention needed
}

4. Monitor Memory Usage

Track memory usage to verify savings:

import "runtime"

func trackMemory() {
    var m runtime.MemStats
    runtime.ReadMemStats(&m)

    fmt.Printf("Heap: %v MB\n", m.HeapAlloc / 1024 / 1024)
    fmt.Printf("Total Alloc: %v MB\n", m.TotalAlloc / 1024 / 1024)
}

5. Log Retention Policy

Make retention policies visible in logs:

logger.Info("generating text",
    "prompt_size", len(prompt),
    "retention_enabled", retention != nil,
)

Debugging Considerations

Trade-offs

When retention is enabled, you lose access to:

• Raw request payloads (for debugging API calls)
• Raw response payloads (for inspecting provider responses)

Debugging Strategies

1. Conditional Retention:

retention := &types.RetentionSettings{
    RequestBody:  types.BoolPtr(!debugMode),
    ResponseBody: types.BoolPtr(!debugMode),
}

2. Separate Debug Requests:

if needsDebugging {
    // Make a debug request without retention
    debugResult, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
        Model:  model,
        Prompt: prompt,
        // No retention - keep raw data
    })
    logDebugInfo(debugResult)
}

// Normal request with retention
result, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:                 model,
    Prompt:                prompt,
    ExperimentalRetention: retention,
})

3. Log Before Generation:

logger.Debug("about to generate", "prompt", prompt)

result, _ := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:                 model,
    Prompt:                prompt,
    ExperimentalRetention: retention,
})

logger.Debug("generation complete", "text", result.Text)

Compliance and Privacy

GDPR Compliance

Retention settings help with GDPR compliance:

• Don't store personal data longer than necessary
• Reduce attack surface for data breaches
• Simplify data retention policies

// For EU customers, always exclude bodies
if customer.Region == "EU" {
    retention = &types.RetentionSettings{
        RequestBody:  types.BoolPtr(false),
        ResponseBody: types.BoolPtr(false),
    }
}

Audit Trails

Keep audit logs without storing full content:

auditLog := AuditEntry{
    Timestamp:    time.Now(),
    UserID:       user.ID,
    Model:        model.ModelID(),
    TokensUsed:   result.Usage.GetTotalTokens(),
    FinishReason: result.FinishReason,
    // No sensitive content stored
}

Examples

See the retention examples for complete working code:

• Basic Usage - Simple examples of retention settings
• Memory Benchmark - Measure memory savings

API Reference

See RetentionSettings API Reference for complete type documentation.

Related Topics

• Context Management
• Error Handling
• Testing

Summary

Retention settings provide:

• ✅ 50-80% memory reduction
• ✅ Privacy protection
• ✅ Better garbage collection
• ✅ No performance overhead
• ✅ Backwards compatible (opt-in)
• ✅ Production-ready

Enable retention settings in your production applications to reduce memory usage and improve performance.