Skip to main content
View as MarkdownOpen in ChatGPTOpen in Claude

Groq Provider

Groq provides the fastest AI inference in the world using custom Language Processing Units (LPUs). Offers speeds up to 10x faster than traditional GPUs, perfect for latency-sensitive applications.

Setup​

Installation​

import (
    "github.com/digitallysavvy/go-ai/pkg/ai"
    "github.com/digitallysavvy/go-ai/pkg/providers/groq"
)

Configuration​

provider := groq.New(groq.Config{
    APIKey: os.Getenv("GROQ_API_KEY"),
})

model, err := provider.LanguageModel("llama-3.3-70b-versatile")
if err != nil {
    log.Fatal(err)
}

Groq has its own provider package (pkg/providers/groq); it does not support embeddings, image generation, or speech synthesis — EmbeddingModel, ImageModel, and SpeechModel all return errors. LanguageModel defaults to "mixtral-8x7b-32768" when the model ID is empty.

Get API Key​

  1. Sign up at console.groq.com
  2. Create API key
  3. Set environment variable:
export GROQ_API_KEY=gsk_...

Available Models​

Language Models​

Model IDContextTokens/SecBest For
llama-3.3-70b-versatile128K300+Best balance
llama-3.2-90b-vision-preview128K250+Multimodal
llama-3.1-70b-versatile128K300+General purpose
mixtral-8x7b-3276832K500+Very fast
gemma2-9b-it8K800+Ultra-fast

All models FREE in public beta with rate limits.

Provider-Specific Features​

Ultra-Fast Streaming​

Groq excels at real-time streaming:

stream, err := ai.StreamText(ctx, ai.StreamTextOptions{Model: model, Prompt: "Write a story"})
if err != nil {
    log.Fatal(err)
}
defer stream.Close()

// Tokens arrive at 300-800 tokens/second
for chunk := range stream.Chunks() {
    fmt.Print(chunk.Text)
}

Tool Use​

Function calling with high-speed inference:

tools := []types.Tool{
    {
        Name:        "calculate",
        Description: "Perform calculation",
        Parameters: map[string]interface{}{
            "type": "object",
            "properties": map[string]interface{}{
                "expression": map[string]string{"type": "string"},
            },
        },
    },
}

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "What is 234 * 567?",
    Tools:  tools,
    StopWhen: []ai.StopCondition{ai.IsStepCount(5)},
})

JSON Mode​

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{
    Model:  model,
    Prompt: "Extract structured data from: John, age 30",
    ResponseFormat: &provider.ResponseFormat{Type: "json_object"},
})

Examples​

Basic Text Generation​

package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "time"

    "github.com/digitallysavvy/go-ai/pkg/ai"
    "github.com/digitallysavvy/go-ai/pkg/providers/groq"
)

func main() {
    ctx := context.Background()
    provider := groq.New(groq.Config{
        APIKey: os.Getenv("GROQ_API_KEY"),
    })

    model, err := provider.LanguageModel("llama-3.3-70b-versatile")
    if err != nil {
        log.Fatal(err)
    }

    start := time.Now()
    result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{Model: model, Prompt: "Explain Groq LPU technology"})
    if err != nil {
        log.Fatal(err)
    }

    elapsed := time.Since(start)
    fmt.Println(result.Text)
    fmt.Printf("\nCompleted in %v (%.0f tokens/sec)\n",
        elapsed,
        float64(result.Usage.GetOutputTokens())/elapsed.Seconds())
}

Real-Time Chat Application​

func streamChat(ctx context.Context, model provider.LanguageModel, message string) {
    stream, err := ai.StreamText(ctx, ai.StreamTextOptions{
        Model:  model,
        Prompt: message,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer stream.Close()

    // Ultra-fast streaming for real-time UX
    for chunk := range stream.Chunks() {
        fmt.Print(chunk.Text)
        // No perceptible delay between tokens
    }
    fmt.Println()
}

Best Practices​

  1. Model Selection

    • Use llama-3.3-70b for best quality
    • Use mixtral-8x7b for speed
    • Use gemma2-9b for ultra-fast responses

  2. Leverage Speed

    • Use streaming for real-time UX
    • Build interactive applications
    • Enable instant feedback loops

  3. Cost Management

    • Free during beta period
    • Monitor rate limits
    • Plan for future pricing

Rate Limits​

Free Tier​

ModelRPMRPDTokens/Min
Llama 3.3 70B3014,4006,000
Mixtral 8x7B3014,4005,000
Gemma 9B3014,40015,000

Error Handling​

result, err := ai.GenerateText(ctx, ai.GenerateTextOptions{Model: model, Prompt: prompt})
if err != nil {
    if strings.Contains(err.Error(), "rate_limit") {
        log.Println("Rate limited - wait before retry")
        time.Sleep(time.Second * 2)
    }
}

Transcription​

provider.TranscriptionModel(modelID) (default "whisper-large-v3-turbo") wraps Groq's Whisper endpoint:

transcriptionModel, err := provider.TranscriptionModel("whisper-large-v3-turbo")
if err != nil {
    log.Fatal(err)
}

result, err := ai.Transcribe(ctx, ai.TranscribeOptions{
    Model: transcriptionModel,
    Audio: audioBytes,
})

Assistant Content With Tool Calls​

Groq sends the assistant message's text content verbatim on tool-call turns, including an empty string, rather than null — matching the TypeScript SDK. See OpenAI-compatible providers: Assistant Content Serialization for how this compares across providers.

Workflow Serialization​

Groq transcription models can cross a workflow boundary with providerutils.SerializeModel / DeserializeModel (language models could already be serialized). See Provider Serialization for the mechanism.

See Also​