<aside> 📄

Paper InPars: Data Augmentation for Information Retrieval using Large Language Models

Authors Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Rodrigo Nogueira

Venue arXiv 2022.02

Links arXiv

Keywords Data Augmentation LLM Query Generation Reranker Few-shot Prompting

</aside>

TL;DR

Use GPT-3 few-shot prompting (3 document–query examples) to generate a query for each target-domain document, keep the top-k most likely pairs, and use them as positives to train a reranker. Models trained only on this synthetic data beat BM25 and self-supervised dense retrievers; combining with MS MARCO improves zero-shot transfer.

1. Problem Definition

Task: Data augmentation for retrieval.

• MS MARCO's scale enabled zero-shot transfer, but not all IR tasks and domains benefit equally from one dataset; domain-specific training data generally helps.
• LLMs like GPT-3 are too expensive to use directly as retrievers or rerankers.

2. Contributions

1. A method to adapt large LMs to IR tasks by using them as synthetic data generators rather than as the model at inference time.
2. Unsupervised, it largely outperforms recent methods; combined with supervised fine-tuning, it reaches SOTA on two of three transfer datasets.

3. Method

1. Prompt GPT-3 with 3 (document, query) examples followed by a target document $d$, and generate a query.
2. Score the generated query by its log-probability and select the top-k pairs.
3. Use the selected (query, document) pairs as positives (with BM25-sampled negatives) to train a monoT5 reranker.

4. Findings

• Rerankers trained solely on InPars data outperform BM25 and recent self-supervised dense retrieval methods.
• Training on both supervised (MS MARCO) and synthetic data gives better zero-shot transfer than supervised data alone.