Is It Time to Rethink LLM Pre-Training? with Aditi Raghunathan - #747
Published:Sep 16, 2025 18:08
•1 min read
•Practical AI
Analysis
This article from Practical AI discusses the limitations of Large Language Models (LLMs) and explores potential solutions to improve their adaptability and creativity. It focuses on Aditi Raghunathan's research, including her ICML 2025 Outstanding Paper Award winner, which proposes methods like "Roll the dice" and "Look before you leap" to encourage more novel idea generation. The article also touches upon the issue of "catastrophic overtraining" and Raghunathan's work on creating more controllable and reliable models, such as "memorization sinks."
Key Takeaways
- •The article discusses limitations of LLMs in generating novel ideas.
- •It highlights research on methods like "Roll the dice" and "Look before you leap" to improve LLM creativity.
- •The article touches upon the issue of "catastrophic overtraining" and approaches for creating more controllable LLMs.
Reference
“We dig into her ICML 2025 Outstanding Paper Award winner, “Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction,” which examines why LLMs struggle with generating truly novel ideas.”