Chunking video for multimodal RAG

Deep diveSelf-paced

Chunking video for multimodal RAG

Everything in the multimodal-RAG lesson gets harder the moment the document is a video. Chunking is the one step in any RAG pipeline that is genuinely irreversible - if the answer to a question spans a boundary you drew, no reranker recovers it and no bigger model saves you. Video makes this worse because it is not one signal: speech, on-screen text, visual content, and structural metadata all arrive on the same timeline at different rates, with no agreement about where one idea stops and the next begins.

Two independent decisions, not one

Most confusion here comes from treating chunking as a single choice. It is two: where do you cut (the boundary policy - what counts as one retrievable unit) and what goes in each chunk (the payload policy - what gets embedded, indexed, and shown to the generator). You can pair scene-detected boundaries with a text-only payload, or fixed windows with native video embeddings. Separating the two turns a mess into a small grid of clear tradeoffs.

Where to cut

Fixed-time windows

Build this first

Uniform 30-60s slices with 5-10s overlap. Right when the visual channel is decorative (talking-head, single-speaker). Cheap and predictable, but blind to speech structure - it cuts through the middle of explanations.

Visual boundary detection

Cut where the picture changes (PySceneDetect). Right for edited content - film, ads, product videos - where a human editor already made cut decisions. A static-camera lecture returns one scene; a fast-cut video returns hundreds of unusable fragments. Always enforce a minimum scene length.

Transcript-semantic

Highest-value default

Chunk the ASR transcript on topic drift or embedding-similarity drops, then project back onto the timeline. The right default for lectures, meetings, and webinars, where meaning lives in speech. Clean disfluency before you embed, and use word-level timestamps - segment-level timestamps drift and drift compounds into citation errors.

Reconciled hybrid

What actually ships

Compute visual and transcript boundaries independently; semantic boundaries lead, visual boundaries snap them into place when they agree. Clamp every chunk to a min and max length - unbounded semantic chunking produces useless 4-second fragments and diluted 6-minute ones.

Query-guided (deferred)

Frontier, not production

Skip fixed boundaries entirely and decide clip extent at query time from cross-modal similarity. Genuinely interesting research direction - the indexing story and latency profile aren't there yet for a client deliverable.

Tip

Clamp, don't just detect

Whatever detects your boundaries, enforce a min and max chunk length afterward. This one unglamorous rule is usually the single highest-return line of code in a video pipeline.

What goes in the chunk

Text-proxy

Turn each temporal chunk into an enriched text document - ASR transcript, OCR of on-screen text, a VLM caption of a keyframe, detected entities - then reuse your existing text-retrieval stack. Debuggable, and hybrid (BM25 + dense) catches proper nouns and codes dense embeddings smooth away. Ceiling: it can't represent motion, timing, or visual style.

Native video embeddings

Embed the clip itself with a purpose-built video model or pooled CLIP frame embeddings. Wins on visual-similarity and action queries where the speaker never says the thing you're searching for. Loses on exact terminology.

Fused

Most production systems

Two indexes over identical chunk boundaries, merged at retrieval with reciprocal rank fusion. Identical boundaries matter - if the text and visual indexes disagree about chunk extents, fusion breaks and citations stop lining up.

The schema that makes citations work

`video_id` plus `start_s` is the grounding primitive - it is what lets your UI render a jump-to-timestamp link and lets an eval harness check retrieval against a golden set of expected timestamps. Keep the string you embed separate from the string you show the user: the embed text is a constructed artefact (title prepended, OCR noise dropped, disfluency normalised); the display text is the clean transcript. `prev_id` / `next_id` cost nothing to store and unlock the next pattern.

Retrieve narrow, generate wide

Chunk small enough that embeddings stay precise. After retrieval, expand each hit to include its neighbouring chunks before handing context to the generator. This fixes the most common production failure: the answer was 15 seconds past the chunk boundary and the system had no way to see it.

Window expansion at generation time

def expand(hits, store, before=1, after=1):
    seen, out = set(), []
    for hit in hits:
        window = [hit]
        cur = hit
        for _ in range(before):
            if not cur.get("prev_id"): break
            cur = store[cur["prev_id"]]; window.insert(0, cur)
        cur = hit
        for _ in range(after):
            if not cur.get("next_id"): break
            cur = store[cur["next_id"]]; window.append(cur)
        for c in window:
            if c["chunk_id"] not in seen:
                seen.add(c["chunk_id"]); out.append(c)
    return sorted(out, key=lambda c: (c["video_id"], c["start_s"]))

Example

More frames is not more accuracy

The Video-RAG paper found LongVA's Video-MME score actually declines when frame sampling goes from 128 to 384 frames (52.6% -> 51.8%); their own ablation peaked at 32 sampled frames. Retrieval precision beats brute-force context - the whole argument for chunking properly instead of widening the window.

How to know it's working

Temporal IoU against a golden set - 100-200 questions with gold time spans; measure Recall@1 at IoU 0.3/0.5/0.7. The gap between IoU@0.3 and IoU@0.7 is a direct readout on boundary quality.
RAGAS on the generated answer - faithfulness, relevance, context precision/recall. Useful, but downstream and noisy - don't debug chunking here.
Boundary-crossing rate - the fraction of gold answer spans split across two or more chunks. Isolates chunking from every other variable; drive it down and everything else improves for free.

Two free signals worth defaulting to

For presentation-style content, a slide change is a near-perfect semantic boundary and trivially detectable as a large luminance shift - if your corpus is slide-based, use it as the primary cut. For multi-speaker content, speaker turns from diarisation are the second-best signal; merge adjacent short turns so a question and its answer stay in one chunk rather than splitting on every turn.

Watch out

Common mistakes

Chunks with no standalone meaning ('and that's why it matters') - enforce a minimum length and expand at generation instead.
Segment-level timestamps instead of word-level - the drift compounds into citation errors across an hour.
Never sanity-checking scene count against duration - a single static-camera lecture silently returns one scene.
Text and visual indexes built on different boundaries - fusion breaks and citations disagree.
Tuning chunking against end-to-end answer quality instead of temporal IoU and boundary-crossing rate directly.

Go deeper (optional)