Skip to content
Blog

Multi-Modal RAG for Video: Using Gemini 3 to Build Searchable 4K Video Knowledge Bases

Transcription-only video search throws away the pixels. Learn to embed video natively with Gemini Embedding 2, chunk with overlap, truncate with Matryoshka, and return trimmed clips on a text match.

Published on • August 2, 2026

AI Assistant

Video is where text-only RAG goes to fail. Transcribe the speech and you keep the words — but you lose the red truck at the stop sign, the whiteboard diagram, the UI error state flashing on screen. Multi-modal RAG for video fixes this by embedding the video itself into the same vector space as text queries, so you search over what happens in the footage, not just what’s said.

In this tutorial, you will learn to build a searchable 4K video knowledge base: chunk a source video with overlap, embed each chunk natively with Gemini Embedding 2, truncate the vectors with Matryoshka Representation Learning, and return a trimmed clip when a text query matches.

Why Text-Pipeline RAG Fails for Video

  1. Transcription loss. ASR keeps the words, drops tone, emphasis, and visual meaning.
  2. Frame captioning is a text middleman. A captioner hallucinates descriptions, and retrieval inherits those errors.
  3. Time is lost. Text chunks don’t know when in the video an event happens, so you can’t return a trimmed clip.

Native audio understanding and native multimodal understanding outperforms text-based alternatives like ASR or captioning. — Gemini Embedding 2 paper (https://arxiv.org/html/2605.27295)

The Native Multimodal Embedding Path

Gemini Embedding 2 (GA via the Gemini API, April 2026) is the first embedding model in the Gemini API that maps text, images, video, audio, and documents into a single embedding space, supporting over 100 languages. A single call accepts up to 8,192 text tokens, 6 images, 120 seconds of video, 180 seconds of audio, and 6 pages of PDFs.

Because all modalities live in one vector space, a text query like “red truck at a stop sign” is directly comparable to a 30-second clip at the vector level — no transcription, no caption middleman. That’s what makes sub-second semantic search over hours of footage practical.

Gemini Embedding 2 is the first embedding model in the Gemini API that maps text, images, video, audio, and documents into a single embedding space, supporting over 100 languages. — Google Developers Blog (https://developers.googleblog.com/building-with-gemini-embedding-2/)

Task prefixes

For agentic retrieval, prefix your content at both index and query time to optimize the embedding for the task:

  • Retrieval: "task: retrieval | query: {content}"
  • Search result: "task: search result | query: {content}"

The Pipeline

flowchart LR
    A["4K source video"] --> B["Chunking<br/>(overlapping 20–30s)"]
    B --> C["Embed each chunk<br/>(gemini-embedding-2)"]
    C --> D["Vector store<br/>(ChromaDB / FAISS / Qdrant)"]
    E["Text query"] --> F["Embed query"]
    F --> G["Top-k similarity search"]
    D --> G
    G --> H["Trim clip from source"]
    H --> I["Result: timestamped clip"]

Step 1 — Chunk with overlap

Split the mp4 into overlapping chunks. Overlap matters: if an event spans two chunks, the overlapping window is what lets retrieval find it. Scene-detection-based chunking beats fixed windows when events align with cuts.

If an event spans two chunks, the overlapping window helps but isn’t perfect. Smarter chunking (e.g. scene detection) could improve this. — SentrySearch (https://github.com/pablogarrone/sentrysearch)

Step 2 — Embed natively

from google import genai

client = genai.Client(api_key="YOUR_API_KEY")

def embed_video(path: str) -> list[float]:
    with open(path, "rb") as f:
        video = f.read()
    result = client.models.embed_content(
        model="gemini-embedding-2",
        contents=genai.types.Content(
            parts=[genai.types.Part(
                inline_data=video, mime_type="video/mp4")]
        ),
    )
    return result.embeddings.values   # 3072-dim, truncatable

Step 3 — Truncate with Matryoshka (MRL)

The model is trained with Matryoshka Representation Learning (MRL), so you can truncate the default 3072 dimensions — Google recommends 1536 or 768. Keeping the first 768 dims and L2-normalizing cuts storage and distance computation dramatically at corpus scale.

Trained using Matryoshka Representation Learning (MRL), so you can truncate the default 3072-dimensional vectors to smaller dimensions… We recommend 1536 or 768 for highest efficiency. — Google Developers Blog (https://developers.googleblog.com/building-with-gemini-embedding-2/)

import numpy as np

def compress(embedding: list[float], dims: int = 768) -> list[float]:
    v = np.asarray(embedding[:dims], dtype=np.float32)
    return (v / np.linalg.norm(v)).tolist()

Step 4 — Store, search, trim

import chromadb

collection = chromadb.PersistentClient(path="./video_idx").get_or_create_collection(
    "footage", metadata={"hnsw:space": "cosine"}
)

# index
collection.add(
    ids=[str(i)],
    embeddings=[compressed_emb],
    metadatas=[{"chunk": i, "ts_start": ts_start, "path": "clip.mp4"}],
)

# search
hits = collection.query(query_embeddings=[query_emb], n_results=5)

When a chunk wins, trim the original file from ts_start to ts_start + chunk_seconds and return an actual clip — the answer, not a link to an hour of footage.

Cross-Modal Retrieval Works in Every Direction

Because all modalities share one space, you get text→video, image→video, audio→video, and video→document retrieval for free. Haystack exposes this day-zero via GoogleGenAIMultimodalDocumentEmbedder inside a single pipeline.

Because all modalities share the same vector space, you can use this approach to support cross-modal retrieval in any direction — for example text-to-image, image-to-text, audio-to-video, or video-to-document search. — Haystack Blog (https://haystack.deepset.ai/blog/multimodal-embeddings-gemini-haystack)

Agentic Video RAG

For agents, multimodal embeddings power multi-step reasoning: a research agent scans a folder of video lectures, matches a question to the exact segment, and streams an answer grounded in that clip. Task prefixes tuned to the agent’s goal measurably improve retrieval accuracy.

Multimodal embeddings enable AI agents to execute multi-step reasoning tasks, such as scanning hundreds of files to fix a codebase or cross-referencing disparate PDFs, with improved understanding and accuracy. — Google Developers Blog (https://developers.googleblog.com/building-with-gemini-embedding-2/)

Production Considerations

  • Still-frame detection is heuristic. Skip static chunks by comparing sampled-frame JPEG sizes, or index everything.
  • Chunk boundaries define recall. Scene detection beats fixed windows when events align with cuts.
  • Freshness vs. latency. Live streams need incremental ingestion; static footage can be batched.
  • Storage. Truncate to 768 dims and normalize; a 4K corpus stays cheap.
  • Vector stores. ChromaDB, FAISS, Qdrant, Weaviate, Pinecone, and Agent Platform Vector Search are all supported.

The model’s retrieval quality is state of the art: strong zero-shot performance on MSCOCO, Vatex, MSR-VTT, MMTEB, and MTEB-Code, and robust generalization across specialized domains from astronomy to culinary arts.

Gemini Embedding 2 achieves landmark performance on well-known embedding benchmarks like MSCOCO, Vatex and MMTEB, with a particularly significant leap in code retrieval. — Gemini Embedding 2 paper (https://arxiv.org/html/2605.27295)

Conclusion

Multi-modal RAG for video replaces the transcription middleman with native video embeddings in a single vector space. Chunk with overlap, embed with gemini-embedding-2, truncate with MRL to 768 dims, store in any major vector DB, and return trimmed clips on a text match. Task prefixes sharpen retrieval for agents, and cross-modal search works in every direction.

Next steps: add scene-detection chunking, try image→video retrieval for “find me footage like this still,” and wire an agentic layer that cites the exact clip in its answer. “Show me the part where the red truck stops” should actually find the frames.

References

ai rag multimodal video-search embeddings gemini