NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval
The query encoder does not need to see. A 70M text-only student keeps 95% of a 2B teacher's quality at 50x lower CPU latency.
PhD Candidate, Mobile Cloud Computing Group, Aalto University
I am a PhD candidate at Aalto University, supervised by Prof. Yu Xiao, working on Efficient and Reliable Visual Document Intelligence. I am also interested in general multimodal information understanding. I obtained a dual M.Sc. in Data Science from KTH and Aalto University with honours, and I am a Kaggle Competitions Master.
zhuchenyang.liu@aalto.fi
Making vision-language document retrieval cheap enough to deploy, and asking what those systems ground their answers on and what their indices leak.
The query encoder does not need to see. A 70M text-only student keeps 95% of a 2B teacher's quality at 50x lower CPU latency.
Multi-vector distillation does not need documents at all. Optimal transport aligns student and teacher query tokens, keeping 95% of teacher quality at 26x faster encoding.
NanoVDR shrank only the query encoder. DistilVDR distills the whole retriever end-to-end from a single 8B teacher, keeping most of its quality at 527M with a 15x smaller index.
A document's structure lives in the middle layers, not the final one. Pruning by that signal removes over 90% of the index without any training.
Reranking can be one multiple-choice question over all candidates instead of a yes/no score per candidate. On MMEB-V2 that choice alone beats per-candidate scoring.
A multi-vector index can be inverted back into readable pages. Shuffling the vectors does not stop it, but pooled indices stay substantially harder to invert.
With a language-cue interface, we find the grounding bottleneck is the coordinate output, not the model's ability to locate the evidence.
Adding text recovers instruction understanding but breaks diagram-to-video alignment. Scaling up does not help, because the bottleneck is the visual encoder.
E-textile design, wearable sensing and human motion, mostly with collaborators.