Can Text Encoders be Deceived by Length Attack?

Albeit de facto to use in training dense retrieval models, we observe that contrastive learning is prone to length overfitting, making it vulnerable to adversarial length attacks. We examine the behaviour of this phenomenon and propose an editing method to mitigate this problem. We find that our method can effectively improve the robustness of models against length attacks. Its effectiveness can be attributed to reduced length information in the embeddings, more robust intra-document token interaction