Counterfactual LLM Framework Measures Rhetorical Style in ML Papers

This paper introduces a novel framework for quantifying rhetorical style in machine learning papers, addressing the challenge of distinguishing between genuine empirical results and mere hype. The use of counterfactual generation with LLMs is innovative, allowing for a controlled comparison of different rhetorical styles applied to the same content. The large-scale analysis of ICLR submissions provides valuable insights into the prevalence and impact of rhetorical framing, particularly the finding that visionary framing predicts downstream attention. The observation of increased rhetorical strength after 2023, linked to LLM writing assistance, raises important questions about the evolving nature of scientific communication in the age of AI. The framework's validation through robustness checks and correlation with human judgments strengthens its credibility.

• •LLMs can be used to quantify rhetorical style in research papers.
• •Rhetorical framing, especially visionary framing, impacts the attention a paper receives.
• •The use of LLM writing assistance is correlated with increased rhetorical strength in papers.