LLM Gateways: Unified Routing, Quotas, and Observability for AI Traffic

As your AI applications grow, managing LLM calls across multiple providers becomes unsustainable. Different SDKs, authentication patterns, request formats, and error types for every model create operational chaos. LLM Gateways solve this by providing a unified interface to all providers, with built-in routing, quotas, and observability.

LiteLLM is the leading open-source AI Gateway, giving you a single interface to call 100+ LLM providers using the OpenAI format.

Why an LLM Gateway

Problem	Without Gateway	With Gateway
Provider switching	Rewrite code per provider	Single API, swap providers
API key management	Keys scattered across services	Centralized key vault
Cost tracking	Manual per-provider billing	Unified spend dashboard
Rate limiting	Per-service implementation	Gateway-level enforcement
Observability	Fragmented logging	Centralized traces
Fallbacks	Manual retry logic	Automatic failover

Setting Up LiteLLM

Installation

# Python SDK
pip install litellm

# Gateway proxy server
pip install 'litellm[proxy]'

# Docker
docker pull ghcr.io/berriai/litellm:main

Quick Start

Start the gateway with a single command:

litellm --model gpt-4o

This creates a proxy server at http://0.0.0.0:4000 that forwards to OpenAI. Add more providers:

litellm --model gpt-4o,claude-sonnet-4-20250514,gemini-2.0-flash

Client Integration

Use the OpenAI SDK to call any provider:

import openai

client = openai.OpenAI(
    api_key="anything",  # Gateway handles auth
    base_url="http://0.0.0.0:4000",
)

# Call OpenAI
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Call Anthropic (same interface)
response = client.chat.completions.create(
    model="claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Call Gemini (same interface)
response = client.chat.completions.create(
    model="gemini-2.0-flash",
    messages=[{"role": "user", "content": "Hello!"}],
)

Python SDK

For direct library integration:

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Anthropic
response = completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello!"}],
)

Routing Strategies

Load Balancing

Distribute traffic across multiple instances:

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-4o",
            "litellm_params": {
                "model": "openai/gpt-4o",
                "api_key": "key-1",
            },
        },
        {
            "model_name": "gpt-4o",
            "litellm_params": {
                "model": "openai/gpt-4o",
                "api_key": "key-2",
            },
        },
    ],
    routing_strategy="simple-shuffle",
)

Fallbacks

Automatic failover when a provider is unavailable:

router = Router(
    model_list=[
        {
            "model_name": "primary",
            "litellm_params": {"model": "openai/gpt-4o"},
        },
        {
            "model_name": "fallback",
            "litellm_params": {"model": "anthropic/claude-sonnet-4-20250514"},
        },
    ],
    fallbacks=[{"primary": ["fallback"]}],
)

Cost-Based Routing

Route to the cheapest provider for each task:

router = Router(
    model_list=[
        {
            "model_name": "gpt-4o-mini",
            "litellm_params": {"model": "openai/gpt-4o-mini"},
            "cost_per_token": {"input": 0.15, "output": 0.60},
        },
        {
            "model_name": "claude-haiku",
            "litellm_params": {"model": "anthropic/claude-haiku"},
            "cost_per_token": {"input": 0.25, "output": 1.25},
        },
    ],
    routing_strategy="cost-based",
)

Virtual Keys and Quotas

Virtual API Keys

Create virtual keys for each service:

# In proxy config
virtual_keys = [
    {
        "key": "sk-service-a",
        "models": ["gpt-4o", "gpt-4o-mini"],
        "max_budget": 100.00,  # $100 limit
        "budget_duration": "30d",
        "rpm_limit": 100,  # Requests per minute
        "tpm_limit": 100000,  # Tokens per minute
    },
]

Budget Controls

Enforce spending limits per key or user:

# proxy_server_config.yaml
general_settings:
  master_key: "sk-your-master-key"

litellm_settings:
  drop_params: true
  set_verbose: false

router_settings:
  routing_strategy: "simple-shuffle"
  num_retries: 3
  timeout: 30

Observability

Built-in Logging

LiteLLM logs all requests automatically:

# Configure logging
litellm_settings = {
    "success_callback": ["langfuse", "gcs_bucket"],
    "failure_callback": ["langfuse"],
    "service_callback": ["opentelemetry"],
}

Tracing Integration

Connect to observability platforms:

# Langfuse
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-..."

# OpenTelemetry
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4317"

Cost Tracking

Monitor spending per model and user:

from litellm import completion
from litellm.integrations import CostTracking

cost_tracker = CostTracking()

response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    metadata={"user_id": "user-123"},
)

# Track cost
cost_tracker.track(response)

Gateway Configuration

Docker Deployment

# docker-compose.yml
version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main
    ports:
      - "4000:4000"
    volumes:
      - ./proxy_server_config.yaml:/app/proxy_server_config.yaml
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    command: "--config /app/proxy_server_config.yaml"

Kubernetes Deployment

# helm/litellm/values.yaml
replicaCount: 3
image:
  repository: ghcr.io/berriai/litellm
  tag: main
service:
  type: LoadBalancer
  port: 4000
env:
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: llm-keys
        key: openai

Production Checklist

Component	Configuration
Authentication	Master key, virtual keys per service
Rate limiting	RPM and TPM limits per key
Budget controls	Spend caps and alerts
Fallbacks	Automatic provider failover
Logging	Centralized logging to observability platform
Monitoring	Latency, error rate, cost dashboards
Security	Input validation, PII masking
Deployment	Docker/Kubernetes with health checks

Performance

LiteLLM achieves 8ms P95 latency at 1,000 requests per second. Key optimizations:

• Connection pooling: Reuse HTTP connections to providers
• Async I/O: Non-blocking request handling
• Response caching: Cache identical requests
• Streaming support: Stream responses without buffering

Next Steps

• Read the LiteLLM Documentation for complete configuration reference
• Explore Cost Monitoring for tracking gateway spending
• Learn about Guardrails for gateway-level output validation

An LLM Gateway is the foundation of enterprise AI infrastructure. By centralizing routing, quotas, and observability, you gain control over your AI traffic while maintaining the flexibility to switch providers as needed.