<aside> 📄

Paper Large Dual Encoders Are Generalizable Retrievers (GTR)

Authors Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, Yinfei Yang

Venue EMNLP 2022 · arXiv 2021.12

Links arXiv

Keywords Generalizable Retrieval Scaling Dual Encoder BEIR

</aside>

TL;DR

A common belief is that the dot-product bottleneck makes dual encoders bad at out-of-domain retrieval. GTR keeps the embedding size fixed at 768 and simply scales up the T5 encoder (up to XXL); with multi-stage training, out-of-domain performance on BEIR improves significantly — and it needs only 10% of MS MARCO to reach its best out-of-domain results.

1. Problem Definition

Task: Build a generalized retriever that works across domains.

• Dual encoders trained on one domain often fail to generalize to others.
• Widespread belief: the bottleneck layer — the final score is just a dot product between a query vector and a passage vector — is too limited for out-of-domain generalization.

2. Contributions

• Show that a generalized retriever can be built by scaling up model size while keeping the bottleneck embedding size fixed.
• GTR (Generalizable T5-based dense Retrievers) significantly outperforms existing sparse and dense retrievers on BEIR.

3. Method

• Keep the embedding at 768 dimensions and the query–document interaction a simple dot product.
• Scale the dual encoder (T5 encoder, as in Sentence-T5) from base to XXL.
• Multi-stage training: pre-train on large-scale web question–answer pairs, then fine-tune on MS MARCO.

4. Findings

• Scaling up model size brings significant improvements on many retrieval tasks, especially out-of-domain generalization.
• Most surprisingly, GTR is very data efficient: only 10% of MS MARCO supervised data is needed to achieve the best out-of-domain performance.