<aside> 📄

Paper Dense Passage Retrieval for Open-Domain Question Answering (DPR)

Authors Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih

Venue EMNLP 2020 · arXiv 2020.04

Links arXiv

Keywords Dense Retrieval ODQA Dual Encoder Contrastive Learning

</aside>

TL;DR

With the right training setup, simply fine-tuning two BERT encoders (question and passage) on existing question–passage pairs with in-batch negatives plus one BM25 hard negative beats BM25 by a large margin — no extra pre-training like ICT needed.

1. Problem Definition

Task: Open-domain question answering (retriever + reader).

• Passage retrieval in ODQA was dominated by sparse methods (TF-IDF, BM25).
• Dense retrieval had been explored (e.g., ORQA), but required expensive extra pre-training and did not clearly beat BM25.

2. Contributions

1. With a proper training setup, fine-tuning question and passage encoders on existing question–passage pairs is enough to greatly outperform BM25 (9–19% absolute in top-20 retrieval accuracy).
2. In ODQA, higher retrieval precision translates to higher end-to-end QA accuracy.

3. Method

• Two independent BERT encoders (bi-encoder): one for questions, one for passages. The [CLS] representation (d = 768) is used as the embedding.
• Dot-product similarity between question and passage vectors, enabling maximum inner product search (FAISS) over all passages.
• Contrastive training: maximize the score of the positive passage against negatives.
• Negatives: in-batch negatives (other questions' positives in the same batch) plus BM25-retrieved hard negatives that don't contain the answer.

4. Findings

• Best model: batch size 128, in-batch negatives + 1 BM25 hard negative per question.
• Large improvements over BM25 on NQ, TriviaQA, WebQuestions, and CuratedTREC; combined with a reader, it set new state-of-the-art on multiple ODQA benchmarks.