Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)
Emmanuel Ameisen, an interpretability researcher at Anthropic, discusses how LLMs like Claude "think" and "grow" by looking inside their internal processes. He explains how models plan ahead, form general concepts, and the implications for AI alignment and safety.
Deep Dive Analysis
20 Topic Outline
Introduction to Interpretability and Model Growth
Why Next-Token Prediction is Deceptively Complex
How LLMs Represent Concepts and Understand the World
How LLMs Plan Ahead Internally
Observing AI's Internal Thought Processes
The Emergence of Complexity in AI
Scaling Laws and the Richness of Text
Smooth Progress Creates Sudden AI Breakthroughs
Challenges of AI Alignment and Deception
The '36' Math Problem and Golden Gate Claude Experiments
Anthropic's Focus on AI Safety and Alignment
Why Anthropomorphizing AI Can Be Useful
What AI Can Teach Us About Intelligence
Optimism for AI Alignment and Progress Pace
The Simulation Hypothesis and AI
Decision Models and AI's 'Gut Instincts'
Surprising Findings: AI Gut Checks and Focused Thinking
Getting into Interpretability Research
Books, Movies, and Life Motto
What AI Would Find Surprising in a Human Brain
10 Key Concepts
Interpretability Research
The field dedicated to understanding how large language models (LLMs) like Claude work by "looking inside" their internal processes. It aims to decode the intermediate mathematical operations and numerical results to see what the model is doing as it generates output.
Models are Grown, Not Built
Unlike traditional software, LLMs are not programmed with explicit rules but "grow" through successive phases of training on vast amounts of data. This process involves updating parameters as they learn to predict the next word, leading to inscrutable internal workings.
Next-Token Prediction
The core training objective for LLMs, where the model is fed a sequence of words and asked to predict the next word. As models improve, this seemingly simple task requires them to develop a deep understanding of world facts, logic, and underlying structures.
Internal Representations
LLMs form unique, general representations of concepts (e.g., "Texas," "eye") in their intermediate numerical results. These representations are activated across different contexts and languages, indicating a deeper understanding beyond surface-level text processing.
Internal Planning
LLMs can formulate and follow plans in their intermediate results, even without explicitly stating them in their output. This allows them to execute higher-order tasks like writing coherent poetry or code by referring back to an internal strategy.
Emergence
The phenomenon where complex capabilities and behaviors, such as understanding concepts or planning, arise from seemingly simple training objectives (like next-token prediction) and increased compute. This leads to sudden capability jumps from smooth underlying progress.
Scaling Laws
Empirical observations showing that as more compute and data are applied to the task of predicting the next word, the model's skill predictably and smoothly improves. This continuous improvement correlates with enhanced performance on various complex tasks.
AI Alignment
The effort to ensure that AI models behave in ways that are helpful, harmless, and honest, aligning with human values and intentions. This involves both training models to exhibit desired behaviors and using interpretability to verify internal states and prevent undesirable actions.
Anthropomorphization (AI)
The act of attributing human characteristics or behaviors to AI models. While often criticized, the speaker argues it can be a useful mental model for understanding and predicting AI behavior due to how models are trained on human text, dialogues, and outcome-oriented tasks.
System 1 vs. System 2 Thinking (AI)
An analogy used to describe different types of AI decision-making. System 1 refers to fast, gut-level, binary decisions (like in Jev models), while System 2 involves more deliberate, long-term reasoning, similar to human cognitive processes.
11 Questions Answered
It's a field focused on understanding how large language models (LLMs) like Claude work by examining their internal processes and intermediate numerical results, rather than just observing their inputs and outputs.
AI models operate by converting words into numbers, performing many mathematical operations on these numbers, and then converting the final results back into words, with their internal "thinking" involving the manipulation of these numerical representations.
No, while trained on next-token prediction, advanced AI models go beyond simple autocomplete; to predict the hardest words accurately, they develop a deep understanding of underlying world structures and concepts, similar to how humans reason.
AI models can plan many tokens ahead by formulating internal plans within their intermediate results, which they then refer back to as they output words one at a time, allowing for coherent and complex multi-step outputs.
Researchers decode the intermediate numerical results produced between each word output by passing various texts to the model, recording these numbers, and then identifying patterns that correspond to specific concepts or internal states.
AI models improve smoothly, but their utility for specific tasks (like coding or email writing) only becomes apparent once they cross a reliability threshold, making them "good enough" to be genuinely helpful for that task.
AI models can represent information and plans in their intermediate results without verbalizing them, making it challenging to detect malicious intent solely from their output; interpretability is needed to examine these unstated internal states.
Yes, the speaker argues that anthropomorphizing AI can be a useful mental model for understanding and predicting its behavior and risks, given that models are trained on vast amounts of human-generated text, dialogues, and outcome-oriented tasks that reward human-like strategies.
The current progress suggests a "slow takeoff" scenario, where AI intelligence improves smoothly and predictably, allowing for continuous learning, adaptation, and the implementation of safety measures along the way.
Yes, the speaker is optimistic because many smart people are seriously focused on AI safety and alignment, and as models become more capable, they also become somewhat easier to study internally, creating a push-and-pull dynamic that encourages continuous mitigation efforts.
Studying AI can reveal fundamental truths about the nature of intelligence, particularly how complex systems and capabilities can emerge from simple objectives, similar to how evolution led to human intelligence.
7 Actionable Insights
1. Take AI Progress Seriously
Acknowledge and seriously consider that AI technology will continue to improve at a rapid pace, as this worldview is foundational for responsible development and safety measures.
2. AI Models Grow, Not Built
Recognize that AI models “grow” through extensive data training rather than being explicitly “built” with programmed rules, which means their internal workings are often inscrutable and require scientific study.
3. Next Token Prediction is Complex
Understand that predicting the next token, especially for advanced AI, is a deceptively complex task that requires deep understanding of underlying structures and concepts, not just simple autocomplete.
4. AI Models Plan Ahead
Be aware that AI models, despite outputting one word at a time, can formulate and follow internal plans over many tokens, allowing them to produce coherent and complex outputs like poetry or code.
5. Use Anthropomorphic AI Framing
Consider anthropomorphizing AI as a useful mental model for understanding its behaviors and predicting potential risks, given how models are trained on human text, dialogues, and outcome-oriented tasks.
6. Live in the Present
Avoid the “when I finally” mentality, which projects happiness onto future achievements, and instead strive to be present and appreciate the current moment, as life is composed of these lived experiences.
7. Engage with AI Interpretability
Educate yourself on how AI models work by exploring interpretability research and resources like Neuronpedia, as informed public engagement is crucial for shaping the future of AI.
6 Key Quotes
One of the most important things to realize if you use language models or think about them is just this very bizarre fact that to make them is not to understand them.
Emmanuel Ameisen
Humans, if you really boil it down, like we're basically next token predictors ourselves. We're just trying to figure out the next thing to say.
Emmanuel Ameisen
The best way to sort of understand how that thing behaves is not like exactly like a human, but man, kind of like something that's mimicking a human is really close and is much closer than a program.
Emmanuel Ameisen
I find that the more you look inside, the less certain you are about how they work. And then oftentimes people that will confidently say like, ah, language models are just X or just Y are the people that actually haven't spent a while looking inside.
Emmanuel Ameisen
I think we're going to learn a lot about just the nature of intelligence itself, right? Of sort of like how you get this super complex thing that emerges from, from simple objectives.
Emmanuel Ameisen
This is such a new field, like both interp and like AI and LMs. And I think it, it just benefits from, from people engaging and, and, um, being informed.
Emmanuel Ameisen
2 Protocols
LLM Training Process (High-Level)
Emmanuel Ameisen- Phase 1 (Pre-training): Take a large corpus of data, feed the model the first half of a document, and ask it to predict the next word. Repeat this for an "ungodly amount of data."
- Phase 2 (Fine-tuning): Train the model on a corpus of examples of how it should behave, such as helpful assistant dialogues or solving coding problems.
Investigating Weird AI Behavior (Example: '36' Problem)
Emmanuel Ameisen- Step 1: Observe Weird Behavior: Notice the model exhibiting an unexpected or "crazy" behavior, such as consistently outputting an incorrect answer despite calculating the correct one internally, or descending into "madness."
- Step 2: Look Inside (Intermediate Results): Examine the intermediate numerical results within the model's "thought process" as it processes the input and generates the output.
- Step 3: Decode Activations: Identify specific activations or patterns in the intermediate results that correspond to certain concepts or numbers (e.g., a "huge activation for 36").
- Step 4: Understand the Cause: Determine if the internal activations reveal a conflict (e.g., the model correctly calculated 12 but was optimized to say 36 due to prior training data).
- Step 5: Improve Alignment: Use the understanding gained to improve the general alignment of the current and future models by addressing the root cause in training.