<aside> 📄

Paper Condenser: a Pre-training Architecture for Dense Retrieval

Authors Luyu Gao, Jamie Callan

Venue EMNLP 2021 · arXiv 2021.04

Links ACL Anthology

Keywords Dense Retrieval Pre-training Architecture

</aside>

TL;DR

Standard pre-trained LMs are not structurally ready to be bi-encoders: their [CLS] token stays dormant through middle layers. Condenser adds a short circuit during pre-training so that MLM prediction in the upper layers must condition on the dense [CLS] representation, forcing it to aggregate information — which helps most in low-data settings.

1. Problem Definition

Task: Transformer-based bi-encoders for retrieval and similarity.

• Dense encoders need lots of data and sophisticated training techniques, and suffer in low-data situations.
• Key reason: standard LMs were never trained to aggregate sentence-level information into a single dense vector.
• Analysis: in a BERT bi-encoder, the [CLS] token remains dormant in many middle layers and only reactivates in the last round of attention. An effective bi-encoder should actively aggregate information of different granularity from the whole text across all layers.

2. Contributions

• Condenser: establishes structural readiness by making LM pre-training actively condition on the dense representation — pre-training towards the bi-encoder structure.
• Low-data: sizable improvements over standard LMs, comparable to strong task-specific pre-trained models.
• High-data: Condenser retrievers optimize more easily, outperforming models trained with complicated techniques using only a single round of negative mining.

3. Method

• Split the encoder into early and late layers, and add a small Condenser head.
• The head receives the late [CLS] representation together with early-layer token representations (a short circuit from early outputs to the head), and performs MLM.

• Because token information from the late layers is cut off, any new information produced in the late backbone can only reach the head through $h_{cls}^{late}$ — so the late [CLS] must aggregate it.