<aside>
📄
Paper No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval
Authors Guilherme Moraes Rosa, Luiz Bonifacio, Vitor Jeronymo, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Rodrigo Nogueira
Venue arXiv 2022.06
Links arXiv
Keywords Zero-shot Retrieval Model Size Re-ranker vs. Dense BEIR
</aside>
TL;DR
Small distilled dense models have become the default for deployment, but this paper shows that parameter count and early query–document interaction drive zero-shot generalization. Bigger models give marginal in-domain gains but much larger out-of-domain gains, and re-rankers clearly beat dense retrievers of similar size. The largest re-ranker (monoT5-3B) reaches SOTA on 12 of 18 BEIR datasets.
1. Problem Definition
Task: Retrieval — find the documents most relevant to a query — evaluated both in-domain and out-of-domain (zero-shot).
- Recent work showed small distilled models compete with much larger ones, so distilled dense models became the go-to choice for latency-constrained deployment.
- Is that conclusion still valid when the target domain differs from the training domain?
2. Contributions
- Empirically show that increasing model size gives marginal gains in-domain but much larger gains in new domains (zero-shot).
- Re-rankers largely outperform dense retrievers of similar size on several tasks.
3. Method
- Train re-rankers (cross-encoders, monoT5 family) and dense retrievers of various sizes on MS MARCO, and evaluate both in-domain (MS MARCO) and zero-shot on BEIR.
4. Findings
- Strong zero-shot effectiveness in new domains can be achieved by increasing the number of parameters, without in-domain fine-tuning.
- Re-rankers are more effective than dense retrievers; early query–document interaction (cross-attention) and more parameters both help zero-shot effectiveness.
- The largest re-ranker reaches SOTA on 12 of 18 BEIR datasets, +3 average points over the previous best.
- In-domain effectiveness is not a good indicator of zero-shot effectiveness.
5. Takeaways