Cost Monitoring for Production LLM Applications

Most teams know the unit price of a model, but almost nobody knows what a single production feature actually costs. “Ask our support bot” sounds cheap at $X per million tokens — until a 200K-token RAG context is injected into every call and the invoice arrives.

In this post, you will learn how to capture token usage at the call site, attribute spend to users and features, emit metrics to a time-series backend, and build alerting that catches cost regressions before the CFO does. We’ll use Python, OpenTelemetry, and the Gemini API, but the patterns apply to any provider.

Why token accounting is hard

LLM providers bill on tokens, not requests. A single “request” spans system prompt, conversation history, retrieved context, and output — and the mix changes per call. Without structured usage capture, your cost data hides inside response objects and never makes it to a dashboard.

The first rule of LLM cost monitoring: capture usage metadata on every call, at the moment the call happens.

Capturing usage at the call site

Most SDKs return token counts in the response. With the Gemini API, the usage is on response.usage_metadata:

from google import genai

client = genai.Client()

response = client.models.generate_content(
    model="gemini-2.5-pro",
    contents="Summarize this incident ticket.",
)

meta = response.usage_metadata
print(meta.prompt_token_count, meta.candidates_token_count, meta.total_token_count)

Building a cheap, consistent CostRecorder

Wrap every call in a recorder that normalizes provider metadata into one schema. This is the abstraction that lets you swap models later without rewriting dashboards:

import time
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class LLMCallRecord:
    model: str
    prompt_tokens: int
    completion_tokens: int
    user_id: str
    feature: str            # e.g. "support-bot", "summarizer"
    request_id: str
    latency_ms: float = 0.0
    cache_read_tokens: int = 0
    started_at: float = field(default_factory=time.time)

class CostRecorder:
    def __init__(self, metrics_backend, price_table):
        self.backend = metrics_backend
        self.price_table = price_table

    def record(self, call: LLMCallRecord) -> None:
        input_price, output_price = self.price_table[call.model]
        cost = (
            (call.prompt_tokens - call.cache_read_tokens) * input_price
            + call.cache_read_tokens * input_price * 0.1   # cached tokens bill ~90% cheaper
            + call.completion_tokens * output_price
        ) / 1_000_000
        self.backend.emit(
            metric="llm.cost",
            value=cost,
            tags={"model": call.model, "user": call.user_id,
                  "feature": call.feature, "request": call.request_id},
        )

Key point: cache-read tokens are billed at a fraction of input price, so tracking them separately is where real savings show up. Gemini’s context caching makes this a first-class dimension, not an afterthought.

Emitting metrics with OpenTelemetry

OpenTelemetry gives you a vendor-neutral metric pipeline. A counter keeps the running total, and a histogram gives you the distribution per feature:

from opentelemetry import metrics

meter = metrics.get_meter("llm.cost")
cost_counter = meter.create_counter(
    "llm.cost", description="Total LLM spend in USD", unit="USD"
)
tokens_histogram = meter.create_histogram(
    "llm.tokens_per_request", unit="tokens"
)

def emit_to_otel(call: LLMCallRecord):
    # cost computed as above
    cost_counter.add(cost, {"feature": call.feature, "model": call.model})
    tokens_histogram.record(
        call.prompt_tokens + call.completion_tokens,
        {"feature": call.feature},
    )

Export the meter to Prometheus or Cloud Monitoring and you get a real dashboard. Add the token histogram and you can spot “this feature suddenly sends 10x more context” before it costs you anything.

Budget alerts and regression gates

Cost monitoring is worthless without thresholds. A pragmatic stack:

• Daily budget per feature → alert at 80%, hard-fail (or route to a cheaper model) at 100%.
• Cost per successful request (CPK) → your unit economics. A spike here means context bloat, not traffic.
• Cache hit rate → if it drops, someone changed prompt construction and your bill jumped.

# prometheus rules
groups:
  - name: llm-cost
    rules:
      - alert: DailyBudgetWarning
        expr: sum(increase(llm_cost_total{feature="support-bot"}[1d])) > 0.8 * 100
        labels: { severity: warning }
      - alert: TokenContextBloat
        expr: avg_over_time(llm_tokens_per_request_bucket{feature="rag"}[10m])
              / 1024 > 128
        labels: { severity: warning }

Putting It All Together

Wire the recorder into your FastAPI middleware so every route gets cost attribution automatically:

from fastapi import FastAPI, Request

app = FastAPI()
recorder = CostRecorder(emit_to_otel, PRICE_TABLE)

@app.middleware("http")
async def llm_cost_middleware(request: Request, call_next):
    response = await call_next(request)
    if llm_calls := getattr(request.state, "llm_calls", None):
        for call in llm_calls:
            recorder.record(call)
    return response

Now the dashboard answers: cost per day, per feature, per user, per model — and the alert fires the moment a prompt change inflates context. The full runnable example is in this gist.

Conclusion & Next Steps

You now have capture, attribution, metrics, and alerting for LLM spend. Next steps: (1) run a weekly “cost regression review” where you diff CPK by feature, (2) add a semantic cache to collapse near-identical prompts, and (3) use Gemini’s context caching on your stable RAG system prompt so the 90% cache discount shows up in your metrics.

References / Sources

• Gemini API pricing and context caching. https://ai.google.dev/gemini-api/docs/pricing
• Gemini API docs. https://ai.google.dev/gemini-api/docs
• OpenTelemetry Python metrics. https://opentelemetry.io/docs/languages/python/