<aside>
📄
Paper Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (ANCE)
Authors Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, Arnold Overwijk
Venue ICLR 2021 · arXiv 2020.07
Links arXiv
Keywords Dense Retrieval Hard Negatives Contrastive Learning
</aside>
TL;DR
Dense retrievers underperform because in-batch negatives are uninformative: they give tiny gradients and slow convergence. ANCE instead mines hard negatives globally from the whole corpus using an ANN index that is refreshed asynchronously as the encoder trains.
1. Problem Definition
Task: Contrastive learning for dense text retrieval.
- Dense retrieval (DR) often underperforms sparse retrieval.
- The authors show theoretically that the bottleneck is the domination of uninformative negatives sampled within the mini-batch, which yield diminishing gradient norms, large gradient variance, and slow convergence.
2. Method: ANCE
- Theoretically analyze why local in-batch negatives are ineffective under common DR conditions.
- Select hard negatives globally from the entire corpus: retrieve the top documents for each query with the current model and use the non-relevant ones as negatives.
- Maintain an ANN index of the corpus that is asynchronously refreshed with checkpoints of the encoder while training continues, so negatives stay "hard" for the evolving model.
3. Findings
- Improves results on web search (MS MARCO), question answering, and in a commercial search engine.
- ANCE dot-product retrieval nearly matches the accuracy of a BERT-based cascade IR pipeline, at much lower cost.
- Empirically, ANCE negatives better approximate oracle importance sampling and improve convergence.
4. Takeaways
<aside>
💭
- The key lesson: for dense retrievers, which negatives you train on matters as much as the architecture.
- The cost is significant — periodically re-encoding the whole corpus to refresh the index. Later work (coCondenser) reaches similar quality with simpler negative mining by improving pre-training instead.
- Hard negatives mined from the corpus can include false negatives (relevant but unlabeled passages), which can hurt training if not filtered.
</aside>
5. Related Papers