> Unfortunately , too few people understand the distinction between memorization and understanding. It's not some lofty question like "does the system have an internal world model?", it's a very pragmatic behavior distinction: "is the system capable of broad generalization, or is it limited to local generalization?"
-- a thread from François Chollet

> by popular demand: a starter set of papers you can read on the topic.

"Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks": https://arxiv.org/abs/2311.09247

"Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve": https://arxiv.org/abs/2309.13638

"Faith and Fate: Limits of Transformers on Compositionality": https://arxiv.org/abs/2305.18654

"The Reversal Curse: LLMs trained on "A is B" fail to learn 'B is A'": https://arxiv.org/abs/2309.12288

"On the measure of intelligence": https://arxiv.org/abs/1911.01547 not about LLMs, but provides context and grounding on what it means to be intelligent and the nature of generalization. It also introduces an intelligence benchmark (ARC) that remains completely out of reach for LLMs. Ironically the best-performing LLM-based systems on ARC are those that have been trained on tons of generated tasks, hoping to hit some overlap between test set tasks and your generated tasks -- LLMs have zero ability to tackle an actually new task.

In general there's a new paper documenting the lack of broad generalization capabilities of LLMs every few days.