Skim-Aware Contrastive Learning for Long Document Representation

This paper addresses the challenge of representing long documents, a common issue in fields like law and medicine, where standard transformer models struggle. It proposes a novel self-supervised contrastive learning framework inspired by human skimming behavior. The method's strength lies in its efficiency and ability to capture document-level context by focusing on important sections and aligning them using an NLI-based contrastive objective. The results show improvements in both accuracy and efficiency, making it a valuable contribution to long document representation.