<aside> 📄

Paper TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Authors Kexin Wang, Nils Reimers, Iryna Gurevych

Venue Findings of EMNLP 2021 · arXiv 2021.04

Links arXiv

Keywords Sentence Embedding Denoising Auto-Encoder Unsupervised Domain Adaptation

</aside>

TL;DR

TSDAE learns sentence embeddings without labels by corrupting a sentence (e.g., deleting words), encoding it into a single fixed-size vector, and forcing a decoder to reconstruct the original sentence from that vector alone. It beats prior unsupervised methods by up to 6.4 points and is a strong domain adaptation and pre-training method — evaluated beyond STS on heterogeneous domains.

1. Problem Definition

Task: Unsupervised learning of sentence embeddings.

• Learning sentence embeddings usually needs lots of labeled data, which most domains lack.
• Prior unsupervised methods were mostly evaluated only on Semantic Textual Similarity (STS), which requires no domain knowledge — so it was unclear whether they generalize to other domains and tasks.

2. Contributions

1. TSDAE, a novel unsupervised method based on denoising auto-encoders.
2. The first evaluation of unsupervised sentence embedding methods on various tasks from heterogeneous domains.
3. TSDAE outperforms other methods as a pre-training and domain adaptation method.

3. Method

• A modified encoder–decoder Transformer trained to reconstruct the clean sentence from a noised input (e.g., word deletion).
• The decoder's cross-attention keys and values are restricted to the sentence embedding only — it has no access to the encoder's token-level contextual embeddings.
• This bottleneck forces the encoder to pack meaningful sentence information into a single vector.

Evaluation settings

• Unsupervised learning: only unlabeled sentences from the target task.