Large Language Models and Emergence: A Complex Systems Perspective (Prof. David C. Krakauer)
Published:Jul 31, 2025 18:43
•1 min read
•ML Street Talk Pod
Analysis
Professor Krakauer's perspective offers a critical assessment of current AI development, particularly LLMs. He argues that the focus on scaling data to achieve performance improvements is misleading, as it doesn't necessarily equate to true intelligence. He contrasts this with his definition of intelligence as the ability to solve novel problems with limited information. Krakauer challenges the tech community's understanding of "emergence," advocating for a deeper, more fundamental change in the internal organization of LLMs, similar to the shift from tracking individual water molecules to fluid dynamics. This critique highlights the need to move beyond superficial performance metrics and focus on developing more efficient and adaptable AI systems.
Key Takeaways
- •True intelligence is defined by the ability to solve novel problems with limited information, contrasting with current AI's reliance on vast datasets.
- •The tech community's understanding of "emergence" in LLMs is superficial; true emergence requires fundamental changes in internal organization.
- •The focus should shift from scaling data to developing more efficient internal representations in LLMs to achieve genuine intelligence.
Reference
“He humorously calls this "really shit programming".”