<aside> 📄

Paper LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval

Authors Canwen Xu, Daya Guo, Nan Duan, Julian McAuley

Venue Findings of ACL 2022 · arXiv 2022.03

Links arXiv

Keywords Unsupervised Retrieval Zero-shot Contrastive Learning Hybrid Retrieval

</aside>

TL;DR

LaPraDoR is a dual-tower dense retriever pre-trained with no supervised data. Iterative Contrastive Learning (ICoL) trains query and document encoders alternately with a cache of past representations as extra negatives, and Lexicon-Enhanced Dense Retrieval (LEDR) multiplies BM25 and dense scores. It reaches SOTA on BEIR among dense models and is 22.5× faster than re-ranking.

1. Problem Definition

Task: Unsupervised training of dense retrievers for zero-shot text retrieval.

2. Contributions

• LaPraDoR, an all-round unsupervised pre-trained dense retriever with SOTA performance on BEIR.
• ICoL, an effective training scheme for retrieval models.
• LEDR, an efficient way to combine BM25 with a dense retriever.

3. Method

3.1 Iterative Contrastive Learning (ICoL)

• Positives: query–query and document–document positives via Dropout as Positive Instance (DaPI) (the same text encoded twice with different dropout masks); query–document positives via ICT.
• Negatives: random sampling.
• Cache queue: to enlarge the number of negatives, keep representations from previous steps in a queue and use them as negatives for the current step. Training the query and document encoders iteratively (freezing one while training the other) keeps cached representations in the same embedding space.

3.2 Lexicon-Enhanced Dense Retrieval (LEDR)

• Combine lexical matching with dense retrieval:

$$ \text{score} = \text{BM25}(q, d) \times \text{Dense}(q, d) $$