Skip to content
Blog

Anisotropy in Embedding Spaces and Its Effect on Retrieval Quality

What anisotropy in embedding spaces is, why all sentences seem 'similar' in raw BERT embeddings, and how it degrades retrieval quality.

Published on • November 22, 2026

AI Assistant

Introduction

If you have ever embedded a corpus with a raw transformer and discovered that every pair of sentences has similarity above 0.7, you have met anisotropy. It is a structural flaw in embedding spaces that quietly hurts retrieval — and it is one of the reasons models like Sentence-BERT exist.

What Is Anisotropy?

An isotropic embedding space is one where vectors are spread evenly in all directions: unrelated texts point in genuinely different directions, so their cosine similarity sits near 0.

An anisotropic space is warped: the embeddings occupy a narrow cone rather than the whole sphere. All vectors share a common dominant direction, so:

  • Random unrelated sentences score 0.6–0.9 cosine similarity
  • The useful variation is compressed into a small slice of the score range
  • Nearest-neighbour search returns “everything is a bit similar” rankings

Why It Happens

Research on BERT-derived embeddings (notably the sentence-embedding literature around SBERT) found that pretrained language models encode a large shared component across all sentences — syntax, register, and the “average sentence” direction — that dominates the geometry. Similarity computed on raw [CLS] or mean-pooled BERT outputs therefore reflects that shared component more than task-relevant meaning.

The practical symptom described in the Sentence-BERT line of work: finding the most similar pair among 10,000 sentences with vanilla BERT required ~50 million pairwise computations (~65 hours), because pairwise scoring was both slow and the embeddings themselves were poor at ranking similarity out of the box.

How Anisotropy Hurts Retrieval

  1. Compressed score range — top-1 and top-100 results differ by 0.02 cosine points; thresholds stop working.
  2. Weak discrimination — hard negatives (a truly irrelevant doc vs. a relevant one) separate barely at all.
  3. Broken hybrid fusion — if vector scores cluster at 0.85–0.95, fusing them with BM25 scores skews the blend.
  4. False confidence — dashboards show “high similarity” for junk pairs.

Mitigations

  • Use a sentence-embedding model trained for similarity (Sentence-BERT and its descendants) instead of raw BERT features.
  • Mean-centre the corpus — subtract the mean embedding before comparison (whitening); this removes the common direction.
  • Normalise vectors to unit length and re-check the score distribution.
  • Hard-negative training / contrastive fine-tuning — teaches the model to spread the space.
  • Post-hoc calibration — z-score or percentile-transform similarity scores before thresholding.

Conclusion

Anisotropy makes an embedding space look healthy (high similarities everywhere) while eroding its ability to discriminate. If your retrieval scores are suspiciously uniform, inspect the geometry: a narrow cone of vectors is the tell. Fix the model or fix the space — then your nearest neighbours become meaningful again.

References