<aside> 📄

Paper Text and Code Embeddings by Contrastive Pre-Training (cpt)

Authors Arvind Neelakantan, Tao Xu, et al., Lilian Weng (OpenAI)

Venue arXiv 2022.01

Links arXiv

Keywords Text Embedding Code Embedding Contrastive Pre-training Scale

</aside>

TL;DR

Contrastive pre-training on unsupervised data at scale — neighboring pieces of internet text as positives, in-batch negatives with very large batches — yields general-purpose text and code embeddings. The same embeddings set a new state-of-the-art on linear-probe classification and show strong semantic search, sometimes competitive with fine-tuned models.

1. Problem Definition

Task: Text (and code) embedding.

• Previous work trained embedding models customized for each use case, with different datasets, objectives, and architectures.
• Can a single model, pre-trained contrastively at scale, serve classification, semantic similarity, and search?

2. Method

• Initialize from GPT-style models of various sizes.
• Positive pairs: neighboring pieces of text from internet data (and (text, code) pairs for code embeddings).
• Negative pairs: in-batch negatives, using very large batch sizes.

3. Findings

• Linear-probe classification (7 tasks): +4% and +1.8% relative over the previous best unsupervised and supervised embeddings.
• Large-scale semantic search: +23.4%, +14.7%, and +10.6% relative over the previous best unsupervised methods on MS MARCO, NQ, and TriviaQA.
• Code search: +20.8% relative over prior work.

4. Takeaways

<aside> 💭

• Same message as Contriever and GTR from a different angle: scale (data, batch size, model) of plain contrastive pre-training can substitute for task-specific supervision.
• The recipe needs very large batches and compute, which is out of reach for most academic setups; GradCache-style tricks partly address the batch-size side. </aside>

5. Related Papers