What was measured and how it was set up

Multi-vector models such as ColBERT keep one vector per document token and score with MaxSim: each query vector picks the best-matching document vector, and the scores are summed. Storage cost grows quickly. According to the Hugging Face post, with 4-bit quantization per coefficient, 200 vectors of 128 dimensions require 12.8 KB per document, or 12.8 GB for one million documents. If the document is reduced to 50 vectors, the same size drops to 3.2 GB.

Token pooling merges similar vectors. Sentence Transformers' HierarchicalTokenPooling class uses Ward hierarchical clustering and replaces each cluster with its mean, which is renormalized. It is applied only to documents, not to queries. The budget is set with pool_factor (roughly one quarter of the vectors for a value of 4), with max_tokens (a per-document cap), or with distance_threshold, a shared merge-distance limit.

The proposed training computes two losses in each batch and averages them: retrieval with the original vectors, and retrieval after compressing the documents to roughly a quarter. Because Ward makes discrete decisions, the method uses a straight-through estimator so that gradients pass from the pooled vectors to the tokens that compose them. The implementation is done with MultiVectorTokenPoolingLoss, which wraps an existing retrieval loss.

Results on NanoBEIR

The author trained two models from answerdotai/ModernBERT-base, each with a new 128-dimension projection: a baseline with only the cached contrastive loss, and one with both losses (no pooling and with a factor of 4). Both ran for one epoch on the same 120,000 MS MARCO triplets, with seed 42, a 300-token limit in training, and expansion of short queries to at least 32 token positions. Evaluation was done on NanoBEIR@13 with average nDCG@10 (%). These are the author's own measurements on their own models, not an independent evaluation.

At pool_factor=4, the average nDCG@10 rises from 49.03 to 51.23, with 74.8% fewer document vectors compared with the no-pooling case. Retention, that is, the ratio of the average nDCG@10 with pooling to the average of the same model without pooling, goes from 94.6% to 97.8%. An improvement also appears without pooling: from 51.84 to 52.39.

The benefits carry over to settings not used in training, such as fixed caps and the adaptive distance threshold. At 29.58 vectors per document on average, adaptive pooling retains 97.3% of the pooling-aware model's no-pooling score, with 84.7% fewer vectors. The conventional model retains 93.3% at the same budget. For the adaptive thresholds, the author calibrated a separate threshold for each encoder and each NanoBEIR corpus so as to match the average vector counts of the caps of 64, 32 and 16.

Average nDCG@10 (%) on NanoBEIR@13 by pooling setting, as reported by the post's author

Pooling settingAverage vectorsConventional modelPooling-aware modelDifference (points)
No pooling193.2251.8452.39+0.55
pool_factor=296.8650.1051.74+1.64
pool_factor=448.6849.0351.23+2.20
pool_factor=632.6247.0749.85+2.78
max_tokens=3229.5848.2550.12+1.87
max_tokens=1615.6046.6149.26+2.65
max_tokens=87.9943.1246.61+3.48
max_tokens=44.0037.2243.00+5.78

Limits, tooling and compression order

The author notes that only the loss differs between the two experiments and that no hyperparameter search was run. It is therefore a comparison of one pair of training runs with a single seed, on one base model and one training set. The post builds on earlier work, such as Stefan Josef's Learn to Pool and LightOn's study on pooling regularization.

On speed, the post mentions optional kernels. The default MaxSim kernel scores without materializing the entire token similarity matrix in memory and requires CUDA and Triton. The Ward kernel runs on CUDA instead of CPU and is 30-150x faster than SciPy, depending on the setting. Training can also run on Hugging Face Jobs, on an A10G GPU with a four-hour limit.

As for the order of index-reduction techniques, the author suggests first quantization to 4, 2 or 1 bit, then token pooling, and third Matryoshka token embeddings to reduce dimensionality. The techniques can be combined, and MultiVectorMatryoshkaLoss can be wrapped with MultiVectorTokenPoolingLoss. The author recommends measuring index size and retrieval quality on your own search engine and your own document corpus.