<aside> 📄

Paper Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Authors Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, Yinfei Yang

Venue Findings of ACL 2022 · arXiv 2021.08

Links ACL Anthology

Keywords Sentence Embedding T5 Contrastive Learning Scaling

</aside>

TL;DR

The first study of sentence embeddings from T5 encoder–decoder models, scaling up to 11B parameters. Encoder-only mean pooling works best for transfer tasks, the encoder–decoder variant sets a new state-of-the-art on STS, and bigger models consistently help.

1. Problem Definition

Task: Sentence embedding.

• T5 performs strongly on many language tasks, but it is unclear how to obtain sentence embeddings from an encoder–decoder model.

2. Contributions

1. Without fine-tuning, encoder-only ST5 models already perform well on sentence transfer tasks.
2. Encoder–decoder sentence embedding models achieve strong STS performance.
3. Contrastive learning is effective for fine-tuning sentence encoders from T5-style models.
4. Training ST5 longer and on more data with a contrastive loss gives consistent improvements on both transfer and STS tasks.
5. A new sentence representation transfer benchmark, SentGLUE, extending SentEval to nine GLUE tasks.

3. Method

Three ways to build a sentence embedding from T5:

1. Encoder first token — the first token representation of the encoder.
2. Encoder mean — average of all token representations from the encoder.
3. Encoder–decoder first — the first decoder output, with the decoder attending to the encoder.