Skip to content
Blog

Streaming Inference with FastAPI and Server-Sent Events

Turn a blocking LLM call into an incremental, observable stream of tokens with FastAPI StreamingResponse and Server-Sent Events — including cancellation and backpressure.

Published on • August 3, 2026

AI Assistant

Streaming is how you turn a minutes-long model call into a visible, responsive one, and it pairs naturally with measurable, observable LLM infrastructure.

The problem: blocking calls

A normal LLM API call is a single POST that returns only when the whole answer is finished:

response = client.models.generate_content(model="gemini-2.5-pro", contents="Explain MD5 to a 5-year-old")
print(response.text)

For long answers the caller stares at a spinner. For agentic loops and chat UIs this is a terrible experience, and it makes “is it working?” unanswerable mid-generation.

Server-Sent Events (SSE): one-way, streaming text

SSE is a lightweight HTTP protocol — a single long-lived connection where the server pushes data: lines as events arrive. It is the right tool when the server pushes to one browser (one-way). It’s simpler than WebSockets and built on plain HTTP.

A token stream from the model looks like:

data: {"text": "Mult"}
data: {"text": "iple "}
data: {"text": "enc"}
...
data: [DONE]

Streaming from Gemini in Python

The Gemini Python SDK supports streaming:

from google import genai

client = genai.Client()
stream = client.models.generate_content_stream(
    model="gemini-2.5-pro",
    contents="Explain MD5 to a 5-year-old in 4 short lines.",
)

for chunk in stream:
    print(chunk.text, end="")

chunk.text arrives progressively. Now wire that into FastAPI.

Building a FastAPI streaming endpoint

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from google import genai
import json

app = FastAPI()
client = genai.Client()

@app.post("/v1/stream")
async def stream_llm(prompt: str):
    async def event_generator():
        stream = client.models.generate_content_stream(
            model="gemini-2.5-pro", contents=prompt
        )
        for chunk in stream:
            # Each token becomes one SSE "data:" line
            yield f"data: {json.dumps({'text': chunk.text})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        event_generator(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

Key points:

  • media_type="text/event-stream" tells the client this is SSE.
  • X-Accel-Buffering: no disables reverse-proxy buffering (Nginx) so chunks reach the client immediately.

Handling cancellation

If the client disconnects, a generator that keeps yielding wastes tokens and money. Detect cancellation and stop the underlying stream:

import asyncio

async def event_generator():
    try:
        stream = await asyncio.to_thread(
            client.models.generate_content_stream,
            model="gemini-2.5-pro",
            contents=prompt,
        )
        for chunk in stream:
            yield f"data: {json.dumps({'text': chunk.text})}\n\n"
            await asyncio.sleep(0)   # let cancellation surface
    except asyncio.CancelledError:
        # stop early: don't keep billing for an abandoned generation
        print("client disconnected, cancelling generation")
        raise

Putting It All Together

The browser consumes it with EventSource:

const es = new EventSource('/v1/stream');
es.onmessage = (e) => {
  if (e.data === '[DONE]') { es.close(); return; }
  appendToChat(JSON.parse(e.data).text);
};

Combine with an observability wrapper (a trace_id span around the generator) — the ref’s “operate AI like production infrastructure” — so you can see tokens/s, abort rate, and cost per stream.

Conclusion & Next Steps

Streaming turns a blocking black-box call into an incremental, observable one. Next: add a text/plain fallback, surface token-count and cost metadata on the [DONE] event, and add cancellation metrics to your dashboard so you know exactly how much an aborted stream cost.

References / Sources

ai fastapi streaming sse llms