Skip to content
Blog

Self-Hosting an LLM Gateway with LiteLLM

One OpenAI-compatible endpoint in front of every model your team uses, with routing, rate limits, spend caps, and auth. Deploy LiteLLM and stop hard-coding provider keys in your services.

Published on • August 8, 2026

AI Assistant

If every service calls its own model provider directly, you have API keys scattered everywhere, no central place to cap spend, and no way to swap models without editing code. An LLM gateway fixes all three: one OpenAI-compatible endpoint your apps already speak, backed by routing, rate limits, budgets, and auth.

Why self-host a gateway (and when not to)?

If your product is consumed by you, or you need air-gapped / on-prem deployments, or you want per-team cost limits and model rotation — self-hosting is the right call. If you have one app and one model, a gateway is overhead you don’t need yet. Start thin; add a gateway when you have multiple models, multiple services, or a spending problem.

Deploy LiteLLM in minutes

The requirements: Docker, a config.yaml, and an admin key. Install:

docker volume create litellm_data
docker run --name litellm \
  -v ./litellm_config.yaml:/app/config.yaml \
  -v litellm_data:/home/llm \
  -p 4000:4000 \
  ghcr.io/berriai/litellm:main-stable

Any service in your fleet now calls http://gateway:4000/v1/chat/completions exactly as it would call OpenAI.

Routed, per-provider config

model_list:
  - model_name: "gpt-class"
    litellm_params:
      model: "openai/gpt-4o"
      api_key: ${OPENAI_API_KEY}
  - model_name: "fast"
    litellm_params:
      model: "gemini/gemini-2.5-flash"
      api_key: ${GEMINI_API_KEY}
  - model_name: "fast"
    litellm_params:
      model: "anthropic/claude-3-7-sonnet"
      api_key: ${ANTHROPIC_API_KEY}

A client asking for "fast" is load-balanced across the two models with the same alias. Failover, retries, and timeouts all live at the gateway, so app code stays provider-agnostic.

Guardrails a dev team needs on day one

  • Cost caps: track spend per team/API key and hard-stop over budget.
general_settings:
  master_key: sk-012...           # required for admin + auth
  database_url: postgresql://...
  max_litellm_cost_per_month: 500 # site-wide budget
  • Rate limiting per key so one noisy job can’t starve prod.
  • Spend logging to a Postgres-backed DB for per-team dashboards.
  • Auth/SSO via admin-managed keys — the app only needs Authorization: Bearer <gateway-key>.

Proxy overhead, the honest number

LiteLLM’s own controlled benchmark puts Python-path per-request overhead around 7.5 ms on a mock upstream, with 8 ms P95 at 1,000 RPS multiplexed on 4 vCPU/8GB instances. Against real model inference latency (hundreds of ms to seconds), that overhead is a rounding error. But it matters when you’re routing embedding/classification traffic at high QPS.

LiteLLM is mid-migration of the hot path to Rust (Q4 2026), targeting sub-1ms overhead and a ~65MB binary — early internal numbers on a mock show ~0.05 ms overhead / 6,782 req/s / 32MB. Future-proof, but today’s Python path is fine for most traffic.

Key practice: read the real x-litellm-overhead-duration-ms header on your own responses instead of trusting anyone’s published number.

Putting It All Together

Wire your app once against http://gateway:4000/v1/ — base_url + api_key in your existing SDK config, nothing else changes. Then:

  • Move new models into the gateway without restarting app code: add an entry to model_list, bump rate limits, done.
  • Point all teams/CI at the same gateway and let IAM/per-key routing replace each system’s private keys.
  • Benchmark your own traffic with a fake provider (network_mock mode) before committing to a sizing.

Conclusion & Next Steps

One endpoint, one key per team, centralized cost and throughput control: LiteLLM turns provider sprawl into something observable and governable. Next: add Postgres-backed logging for a spend dashboard, set per-key spend caps, and, once traffic justifies it, watch the Rust gateway rollout for a lighter deployment — same config.yaml, no migration.

References / Sources

ai gateway litellm llms infrastructure