Multimodal Pipelines: Images, Audio, and Video in LLMs

The fastest way to lose information is to serialize it into text before the model ever sees it. Transcribe the audio, caption the video, describe the image — and you have already picked what matters. Multimodal models accept the raw bytes directly, so the model notices things your abstraction layer threw away.

What “multimodal” means in 2026

gemini-2.5-pro now accepts text, images, video, audio, and PDF as primary inputs, and produces text. For real-time, audio-to-audio, gemini-3.1-flash-live-preview takes audio/video and returns text and audio. Practically, this means one model can do what used to take three separate OCR, ASR, and vision systems — and crucially, jointly reason across them.

The 2026 pricing point matters: text/image/video input for gemini-3.1-flash-lite is ~$0.25/1M tokens, so multimodal is no longer a premium path — it’s the default.

Getting files into the model

The Gemini API gives you four input methods, chosen by file size and reuse:

Method	Best for	Max size	Persistence
Inline data	Small, transient files	100 MB/payload (50 MB PDFs)	None
File API upload	Large or reused files	2 GB per file	48h
GCS URI registration	Files already in GCS	2 GB per file	per-request fetch
External URLs	Public URLs / presigned URLs	100 MB/payload	None

from google import genai

client = genai.Client()
# inline image (base64) or from disk
import PIL.Image

img = PIL.Image.open("dashboard.png")

resp = client.models.generate_content(
    model="gemini-3.1-flash-lite",
    contents=["What anomalies do you see in this dashboard?", img],
)
print(resp.text)

Feed video directly

Video is uploaded once with the File API, then referenced by name:

video = client.files.upload(path="product_demo.mp4")  # returns a video file

resp = client.models.generate_content(
    model="gemini-3.1-flash-lite",
    contents="Summarize this product demo. Give me the 3 key moments with timestamps.",
    config={"files": [video]},
)

The model samples frames and can combine them with any accompanying audio track. A single unstructured segment — a 5-minute demo — that would take a human 10 minutes to caption becomes a structured list in one call.

A pipeline that mixes all three in one call

The value compounds when signals are combined. Example: a meeting recorder that produces minutes-with-links-to-charts.

1. Your meeting platform yields: raw audio (speaker 1), screen-share video (slides + loom of the chart), and the agenda PDF.
2. Post all three to one model with one instruction set.

audio = client.files.upload(path="meeting.wav")
slides = client.files.upload(path="slides.mp4")
agenda = client.files.upload(path="agenda.pdf")

resp = client.models.generate_content(
    model="gemini-3.5-flash",
    contents="Transcribe the meeting, map each action item to the slide it was discussed on (slide #), "
             "and identify the name of the spreadsheet referenced in the agenda.",
    config={"files": [audio, slides, agenda]},
)

One call, three data types, one coherent thread from audio → visuals → document.

Grounding: connect moments to visual evidence

As the model processes a video as frames, you can ground moments via video_metadata clips or a custom frame config. The point: you can ask “where in the clip does the UX show the error, and insert a screenshot into the ticket,” and the model returns frame-level evidence even when it only saw raw minutes.

Putting It All Together

• Track your File API uploads: files are stored 48h and up to 2 GB each, so reference the upload object across edits (multi-turn) to avoid re-transferring bytes on every call.
• Wrap generation in a worker that streams interactions.create, so image/table output streams back to the UI without waiting on a full generateContent resolve.
• Add structured response_format={"type":"json"} when the downstream wants to parse minutes, action items, or taxonomy — combine multimodal input with structured output.

Conclusion & Next Steps

Stop flattening to text. Centralize your genai.Client calls behind a single multimodal_pipeline service; add rich examples for meetings, QA screenshots, and demo movies to your evals. When your traffic justifies it, switch uploads to GCS URI registration to eliminate the 48h TTL and huge re-transfers.

References / Sources

• Gemini API file input methods. https://ai.google.dev/gemini-api/docs/interactions/file-input-methods
• Gemini 3 series model cards. https://ai.google.dev/gemini-api/docs/interactions/interactions-overview
• Gemini AI developer docs — image/audio/video reasoning. https://ai.google.dev/gemini-api/docs