The OpenAI team created an interpretable model which is much more transparent than typical transformers, behave like a "black box."
This is important because such a model helps understand why AI hallucinates, makes mistakes, or acts unpredictably in critical situations.
The new LLM is a sparse transformer: much smaller-simpler than modern LLMs (at level of GPT-1). but goal is not to compete, but to be as explainable as possible.
🟢 How it works?
- the model is trained so that internal circuits become sparse,
- most weights are fixed at 0,
- each neuron has not thousands of connections, but only dozens,
- skills are separated from each other by cleaner and more readable paths.
In usual dense models, neurons are connected chaotically, features overlap, and understanding the logic is difficult.
Here, for each behavior, a small circuit can be identified:
sufficient, because it performs the required function itself,
and necessary, because its removal breaks the behavior.
The main goal is to study how simple mechanisms work to better understand large models.
The interpretability metric here is circuit size,
the capability metric is pretraining loss.
As sparsity increases, capability drops slightly, and circuits become much simpler.
Training "large but sparse" models improves both metrics: the model becomes stronger, and the mechanisms easier to analyze.
Some complex skills, such as variables in code, are still partially understood, but even these circuits allow predicting when the model correctly reads or writes a type.
The main contribution of the work is a training recipe that creates mechanisms
that can be *named, drawn, and tested with ablations*,
rather than trying to untangle chaotic features post hoc.
LIMITS: these are small models and simple behaviors, and much remains outside the mapped chains.
This is an important step toward true interpretability of large AI.