Evaluating LLM-Generated Scientific Summaries
Published:Dec 29, 2025 05:03
•1 min read
•ArXiv
Analysis
This paper addresses the challenge of evaluating Large Language Models (LLMs) in generating extreme scientific summaries (TLDRs). It highlights the lack of suitable datasets and introduces a new dataset, BiomedTLDR, to facilitate this evaluation. The study compares LLM-generated summaries with human-written ones, revealing that LLMs tend to be more extractive than abstractive, often mirroring the original text's style. This research is important because it provides insights into the limitations of current LLMs in scientific summarization and offers a valuable resource for future research.
Key Takeaways
- •Introduces BiomedTLDR, a new dataset for evaluating LLM-generated scientific summaries.
- •LLMs tend to be more extractive than abstractive in generating summaries.
- •Highlights limitations of current LLMs in scientific summarization.
Reference
“LLMs generally exhibit a greater affinity for the original text's lexical choices and rhetorical structures, hence tend to be more extractive rather than abstractive in general, compared to humans.”