<aside> 📄

Paper Unsupervised Dense Information Retrieval with Contrastive Learning (Contriever)

Authors Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, Edouard Grave

Venue TMLR 2022 · arXiv 2021.12

Links arXiv

Keywords Unsupervised Retrieval Contrastive Learning MoCo Few-shot Multilingual

</aside>

TL;DR

Contriever trains a dense retriever with no supervision using contrastive learning on positive pairs built from the same document (random independent cropping works best) and a large pool of negatives via MoCo. It beats BM25 on 11 of 15 BEIR datasets in Recall@100, and is a strong starting point for few-shot and supervised fine-tuning.

1. Problem Definition

Task: Information retrieval.

• Dense retrievers achieve state-of-the-art results where large training sets exist, but do not transfer to new applications without training data and are outperformed there by BM25.

2. Contributions

1. Contrastive learning alone can produce strong unsupervised retrievers — competitive Recall@100 with BM25.
2. In a few-shot setting, the model benefits from a few in-domain examples and beats transferring models trained on large datasets like MS MARCO.
3. As pre-training before MS MARCO fine-tuning, it leads to strong BEIR performance.
4. Strong multilingual and cross-lingual retrieval (e.g., Arabic queries → English documents), impossible with term matching.

3. Method

Train a BERT-base dense retriever with no supervision:

• Building positive pairs from a single document:
  • Inverse Cloze Task — a span as the query, the rest of the document as the key.
  • Independent cropping — two random spans of the same document as query and key.
• Building many negatives:
  • In-batch negatives.
  • Negatives across batches using a MoCo queue with a momentum encoder.
• Best configuration (Contriever): MoCo with random independent cropping.

4. Findings

• For zero-shot and few-shot learning, using the same encoder for queries and documents improves robustness.