<aside> 📄

Paper GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Authors Kexin Wang, Nandan Thakur, Nils Reimers, Iryna Gurevych

Venue NAACL 2022 · arXiv 2021.12

Links arXiv

Keywords Domain Adaptation Zero-shot Retrieval Query Generation Pseudo Labeling

</aside>

TL;DR

GPL adapts a dense retriever to a new domain without labels: generate queries for target-domain passages, mine negatives with an existing retriever, and let a cross-encoder score every (query, positive, negative) triple. Training with MarginMSE on these soft scores fixes QGen's two failure modes — bad generated queries and false negatives — and improves by up to 9.3 nDCG@10.

1. Problem Definition

Task: Unsupervised domain adaptation for dense retrieval.

• Dense retrievers overcome the lexical gap but need large training sets, and their performance degrades severely under domain shift (Thakur et al., 2021, BEIR).
• QGen-style synthetic training helps, but has two problems:
  1. The query generator is imperfect and may produce queries that the passage cannot answer.
  2. Other passages may also be relevant to a generated query (false negatives), especially when training with hard negatives.

2. Contributions

1. GPL, combining a query generator with pseudo labeling from a cross-encoder.
2. Outperforms an out-of-the-box state-of-the-art dense retriever by up to 9.3 nDCG@10 on six domain-specialized datasets; needs less unlabeled target data and trains more robustly than previous methods.
3. An investigation of six recent pre-training methods for domain adaptation of retrieval.

3. Method

1. Query generation: for each passage in the target corpus, generate 3 queries with a T5 encoder–decoder.
2. Negative mining: for each generated query, retrieve 50 negative passages with an existing dense retriever.
3. Pseudo labeling: for each (query, positive, negative) triple, compute the margin with a cross-encoder:

$$ \delta = \text{CE}(Q, P^{+}) - \text{CE}(Q, P^{-}) $$

1. Training: train the dense retriever with MarginMSE loss so that its score margin matches $\delta$.