The Representation Is the Product

Wed, 29 Jul 2026 00:00:00 +0000

Four ideas behind modern multimodal image and video understanding — and why none of them are new.

I gave this talk at Curtin University on 30 July 2026, hosted by the IEEE Western Australia Computer Society Chapter, to a room of students and researchers. A few people asked for the written version, so here it is — shorter than the talk, and with the references I promised.

Meaning lives in the combination

A football clip is not a sequence of frames. It is frames, plus a commentator saying goal, plus a stadium roar, plus a caption someone typed, plus whatever text happens to be on screen. Any one of those channels is ambiguous. Together they are not.

That is the whole reason multimodal understanding exists as a field, and it is worth stating plainly because it breaks the setting most of us learned computer vision in: one image goes in, one label comes out, on a dataset that never changes.

Real systems break three of those assumptions at once. There are several modalities arriving together. The data might be ten thousand items or ten billion. And there is a latency and cost budget that quietly decides what you are actually allowed to ship.

What I want to convince you of is that four reusable ideas carry you across that entire range:

Encoders that turn each modality into a vector
Fusion that decides when to combine those vectors
Retrieval that searches the resulting space
Distillation that makes big-model quality affordable to run

Learn these four and the application becomes, to a first approximation, a choice of index and output head.

These questions are older than you think

Students tend to assume this machinery arrived with CLIP. It didn’t. What arrived was a better answer to questions the field had already been asking for two decades.

Before deep learning, every modality had hand-engineered descriptors: SIFT and HOG for images, MFCCs for audio, TF-IDF bag-of-words for text. Retrieval already looked familiar — Video Google (Sivic and Zisserman, 2003) quantised SIFT descriptors into “visual words” so you could reuse text-search machinery, inverted index and all. Turning an image into a vector and searching it like text is over twenty years old. Cross-modal linking existed too, in the form of canonical correlation analysis, which found linear projections of image and text features that maximally correlate. That is the conceptual ancestor of CLIP. Even the fusion debate was settled vocabulary: early versus late fusion in semantic video analysis is a 2005 paper.

The wall was that features were fixed and generic. All the learning happened in a shallow classifier on top, so every new task meant new feature engineering. People called the resulting problem the semantic gap.

Then features became learned. AlexNet in 2012 moved the design work from the feature engineer to the optimiser, and within a year CNN activations had replaced SIFT as the default descriptor for almost everything — crucially, they transferred. Joint image–text embeddings retired CCA. Captioning and VQA brought encoder–decoder architectures and then attention, which is the seed of the cross-attention fusion we use now. Self-supervision broke the label bottleneck. Transformers took over.

Then alignment, and then reasoning. CLIP in 2021 produced one general-purpose, semantic, aligned space with zero-shot transfer, and retrieval stopped being a per-task engineering project. Shortly after, people bolted that representation onto a language model — frozen encoders, a small projector, an LLM backbone — and got systems that could reason and converse about pixels.

Notice what did not change across any of this. How do I represent each modality? When do I fuse them? How do I search the result? How do I afford to run it? Those four questions are in a 2005 multimedia paper and in a 2026 production system. Only the answers got better.

What did change is which resource is scarce. It used to be good features. Then it was labels. Now it is compute and latency — which is precisely why the fourth pillar exists.

Pillar 1: turn everything into a vector, then search it

Every modality gets mapped to a fixed-length embedding. Vision goes CNN → ViT → hierarchical ViTs like Swin. Audio becomes a spectrogram fed to an encoder such as Whisper, which conveniently gives you both acoustic features and, through its decoder, transcribed text — two modalities for the price of one. Text goes through a transformer encoder, multilingual the moment your data isn’t English.

The mental model that matters: an encoder is a learned function into a space where geometry means meaning. Nearby vectors are similar content.

We don’t hand-design that space, we learn it — usually by contrast. Self-supervised methods like MoCo learn strong visual structure with no labels by pushing an image away from many negatives. CLIP-style training does something different and, for our purposes, more important: it pulls an image and its matching text or audio together, and pushes mismatched pairs apart.

That distinction is the punchline of the section. Visual similarity alone gets you “looks alike.” Alignment to language gets you “means the same.” In one system I worked on, going from self-supervised-only training to self-supervised plus cross-modal alignment moved top-5 recall from roughly 60% to roughly 90%. Same encoder, same data scale — the difference was that the space had been tied to language.

Once everything is a vector, “find similar content” is nearest-neighbour search, and this is the pillar where scale becomes visible:

Thousands to millions of items: exact search is completely fine. A flat index in FAISS, scikit-learn, or honestly just NumPy. Full-dimension vectors, no approximation, set up in an afternoon.
Billions of items, in real time: you need approximate nearest neighbour. HNSW graph indices give low latency at high recall. Keep separate indices per category so search stays relevant.

One trick worth knowing at either scale is Matryoshka representation learning: train a model whose embedding can be truncated — 768 dimensions down to 128 — with very little quality loss. Short vectors for a cheap first pass, long vectors when precision matters. One model, many operating points on the cost curve.

Pillar 2: when do you combine the modalities?

Once each modality is a vector, you have a genuine design choice, and it is an accuracy-versus-modularity trade-off rather than a correctness question.

Pre-fusion combines features before the model reasons, typically with cross-attention. The network can model interactions between modalities directly, which raises the ceiling — but it is heavier, and every modality has to run together.

Post-fusion lets each pipeline produce its own score, then combines at the end with averaging, a small learned head, or explicit decision rules. Modular, parallelisable, easy to add or remove a modality, and cheap. It also misses fine-grained cross-modal cues.

Two questions decide it. Do the modalities need to interact to make this decision? Pre-fuse. Do you need to scale, cache, or swap them independently? Post-fuse. Most production systems answer both, at different levels of the stack: cross-attention inside each model, decision rules between models.

Worth saying out loud, because people underrate it: adding audio to a visual-only classifier in the system I mentioned lifted average precision by about six points. Audio is nearly free to compute when you’re already running Whisper. Multimodal beats unimodal even when the extra signal is cheap.

Pillar 3: MLLMs, and why you probably can’t ship one

A multimodal large language model is modality encoders, plus a projector, plus an LLM backbone. Visual and audio tokens get projected into the language model’s token space so it can reason over them in words.

They are powerful for exactly one reason: world knowledge and higher-level reasoning. A question like does this video actually depict what its caption claims? is not something a from-scratch classifier learns easily. An MLLM more or less already knows.

They are also frequently undeployable. Billions of parameters, per item, over a high-volume stream, under a fixed latency budget. In our case the adapted MLLM was clearly the strongest model offline, at around 80% AP — and ran at roughly 14 queries per second, which is not a production system.

Distillation is the bridge. You train a small, fast student to imitate a big teacher, and the key insight is that you transfer more than labels:

L = L_CE + α · L_KL + β · L_MSE
 ground truth teacher's soft teacher's
 labels output distribution hidden states

The label tells the student what the answer is. The teacher’s soft distribution tells it how confident to be and which wrong answers were nearly right — often called dark knowledge, and it is a far richer signal than a one-hot vector. The hidden-state term makes the student’s internal representation resemble the teacher’s, which works even across different architectures.

Does it help? In our case the same small architecture went from about 71% AP trained normally to about 76% distilled, against an 80% teacher — most of the gap closed, at roughly six times the throughput. That is the entire point of the pillar: MLLM-quality decisions at production latency.

Pillar 4: composing them into something useful

Real systems put the pillars together in a shape you will recognise from web search, recommender systems, and RAG:

A fast head — the distilled classifier or embedder — handles the common, known cases at high precision and low latency. A retrieval-and-rerank tail path embeds the item, retrieves similar references by ANN, and verifies with a multimodal re-ranker. The head is efficient; the tail catches the novel, rare, and long-tail cases a fixed classifier structurally cannot see, because it was never trained on them. In our system the retrieval path added roughly 22% coverage beyond the classifier alone. Then you combine the two with decision rules — post-fusion, at the level of the whole system.

And here is why the representation is the product. Once you have it, the application is mostly a choice of index and head:

You want	What changes
Visual / multimodal search	Nothing — retrieval used directly
Near-duplicate detection	Point the index at your own corpus
Copyright or reference matching	Point the index at a known-content set
Scene understanding, tagging, captioning	Add a lightweight head on the fused features
Recommendation and clustering	Same embeddings, ranked by similarity
Classification, filtering, moderation	The distilled head, used directly

Build the representation once; change the index and the head to change the application.

The pillars are also scale-invariant in concept, even though the engineering differs enormously:

Pillar	Prototype (thousands–millions)	Production (billions, real time)
Encoders	Off-the-shelf open models	Domain pretraining and fine-tuning
Fusion	Concatenate or late-fuse	Cross-attention plus system-level post-fusion
Retrieval	Flat index, full-dimension vectors	ANN, Matryoshka truncation, sharded indices
Distillation	Often skip it — run the big model	Essential, plus quantisation

Building a demo over ten thousand images this weekend? Left column, and you are done in an afternoon. The right column is what you reach for at a billion items with a latency budget. Conceptually it is the same four boxes.

A necessary note on the numbers

Every figure above comes from a single deployed system: Yew, Xu, Saha et al., KDD ‘26 ( ). They are real, measured, and online-validated — and they are properties of that system, not of the general method. Your architecture, data, and domain will move all of them.

One number in particular deserves care. The paper’s headline result is a 6–8% reduction in unwanted-content views in online A/B tests, and that figure is an aggregate across several pipeline upgrades. The per-pipeline components are smaller — around 1.2%, 0.6%, and 2.33% for the individual changes. If you cite the work, don’t let the headline and the components get conflated; they measure different things.

What I would generalise is the direction of the effects, which is consistent across the literature: adding a modality helps, cross-modal alignment helps retrieval a lot, distillation recovers most of a teacher’s quality, and retrieval covers cases classifiers miss. The magnitudes are yours to measure.

Where this is heading

A few threads worth watching, especially if you are looking for a research problem:

Any-to-any models. Unified architectures that take and generate across modalities. The encode-then-classify split is blurring.

Long video and temporal reasoning. Current models are strong on frames and weak on hours. Token compression, memory, and event-level structure are the open engineering problem.

Efficiency as a research question, not an afterthought. How small can a student get before reasoning actually breaks? Nobody has a principled answer. That is a thesis.

Agentic and retrieval-augmented multimodal systems. The model calls tools, searches its own index, and re-checks itself, rather than making one forward pass.

Embodied multimodality. Vision-language-action models, where the output is a policy rather than a label.

Evaluation, which is behind capability. Benchmarks saturate fast and mostly test recognition, not compositional, temporal, or causal understanding. Honest evaluation is undersupplied and you don’t need a GPU cluster to contribute to it.

Four things you could build this month

None of these need proprietary data or a cluster. The pattern is fully reusable with open encoders and open datasets — scale and labels are what’s proprietary.

Semantic search over your own library. An open CLIP model plus a flat index over your photos or videos. One afternoon, no training, and you will have built pillar one end to end.
Add audio and measure the lift. Run Whisper over a video set, fuse the transcribed text with the visual embeddings, and quantify how much retrieval improves. You will believe the multimodal argument much more once you’ve measured it yourself.
Reproduce a fusion ablation. Visual-only versus visual-plus-audio; early versus late fusion; on a small public dataset. This is a publishable-shaped experiment at small scale.
Distil an open MLLM. Pick a task you care about, distil into a small classifier, and plot the accuracy-versus-latency curve. It is the most useful curve in applied ML and almost nobody plots it for their own problem.

The one line to take away

Four pieces: features, fusion, retrieval, distillation. The field has been asking those same four questions since before any of it was deep learning — what changed is that the representation is now learned once and shared, which is why one stack can serve search, deduplication, copyright matching, scene understanding, and moderation.

If you remember one thing: the representation is the product. Everything else is how you search it, and how fast you can run it.

References and pointers

Sivic and Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, ICCV 2003
Snoek, Worring and Smeulders, Early versus Late Fusion in Semantic Video Analysis, ACM Multimedia 2005
Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), 2021
Liu et al., Visual Instruction Tuning (LLaVA), 2023
Kusupati et al., Matryoshka Representation Learning, 2022
Malkov and Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs, 2016
Yew, Xu, Saha et al., KDD ‘26 — (the running example throughout)

Thanks to the IEEE WA Computer Society Chapter and to EECMS at Curtin University for hosting the talk, and to everyone who stayed for the questions.

Computer Vision | Sanjay Saha