<aside> 📄

Paper Improving Passage Retrieval with Zero-Shot Question Generation (UPR)

Authors Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke Zettlemoyer

Venue EMNLP 2022 · arXiv 2022.04

Links arXiv

Keywords Re-ranking Zero-shot Unsupervised Question Likelihood

</aside>

TL;DR

UPR (Unsupervised Passage Re-ranker) re-scores retrieved passages by the likelihood of the question given the passage, computed with an off-the-shelf pre-trained LM (T5/T0, GPT). No task-specific training is needed, it works on top of any retriever, and it improves top-20 accuracy by 6–18% for unsupervised retrievers and up to 12% for supervised ones.

1. Problem Definition

Task: Passage re-ranking for open-domain QA — rescore retrieved passages so the most relevant ones come first.

• Supervised re-rankers need task-specific training data and can fail under distribution shift.

2. Contributions

• UPR, an unsupervised passage re-ranking approach for open-domain retrieval.
• UPR greatly outperforms strong supervised retrievers across many QA datasets when applied on top of them.
• New SOTA in open-domain QA by simply re-ranking passages and running a pre-trained reader — no further changes.

3. Method

• By Bayes' rule, the relevance of passage $z$ to question $q$ can be expressed through the probability of the question given the passage:

$$ p(z \mid q) \propto p(q \mid z)\, p(z) $$

Assuming a uniform prior over the retrieved top-k passages, ranking by $p(z \mid q)$ is equivalent to ranking by $p(q \mid z)$.

• Compute $\log p(q \mid z)$ as the average token log-likelihood of the question, with the LM conditioned on the passage (plus an instruction like "Please write a question based on this passage").