<aside> 📄

Paper Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation (DAR)

Authors Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, Jong C. Park

Venue ACL 2022 · arXiv 2022.03

Links arXiv

Keywords Dense Retrieval Data Augmentation Document Representation

</aside>

TL;DR

Instead of generating new queries, DAR augments document representations directly: it interpolates between document embeddings (mixup-style) and perturbs them (e.g., with dropout), creating extra training signal cheaply. It improves dense retrieval on both labeled and unlabeled documents.

1. Problem Definition

Task: Data augmentation for dense retrieval.

Challenges of dense retrieval:

1. It needs a large amount of labeled training data.
2. It must handle unlabeled documents never seen in training.

Challenges of query augmentation (e.g., QG-based):

1. Infeasible to generate queries for every document in a large collection.
2. Extra training steps are needed to reflect the generated pairs in the model.
3. Suboptimal for handling enormous numbers of unlabeled documents.

2. Contributions

1. Augment documents rather than queries to address insufficient labeled query–document pairs.
2. Two novel document augmentation schemes: interpolation and perturbation of document representations.
3. Strong retrieval performance improvements.

3. Method