Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Oct 18, 2026 Episode Page ↗
Overview

Emmanuel Ameisen, an interpretability researcher at Anthropic, discusses how LLMs like Claude "think" and "grow" by looking inside their internal processes. He explains how models plan ahead, form general concepts, and the implications for AI alignment and safety.

At a Glance
7 Insights
1h 12m Duration
20 Topics
10 Concepts

Deep Dive Analysis

Introduction to Interpretability and Model Growth

Why Next-Token Prediction is Deceptively Complex

How LLMs Represent Concepts and Understand the World

How LLMs Plan Ahead Internally

Observing AI's Internal Thought Processes

The Emergence of Complexity in AI

Scaling Laws and the Richness of Text

Smooth Progress Creates Sudden AI Breakthroughs

Challenges of AI Alignment and Deception

The '36' Math Problem and Golden Gate Claude Experiments

Anthropic's Focus on AI Safety and Alignment

Why Anthropomorphizing AI Can Be Useful

What AI Can Teach Us About Intelligence

Optimism for AI Alignment and Progress Pace

The Simulation Hypothesis and AI

Decision Models and AI's 'Gut Instincts'

Surprising Findings: AI Gut Checks and Focused Thinking

Getting into Interpretability Research

Books, Movies, and Life Motto

What AI Would Find Surprising in a Human Brain

Interpretability Research

The field dedicated to understanding how large language models (LLMs) like Claude work by "looking inside" their internal processes. It aims to decode the intermediate mathematical operations and numerical results to see what the model is doing as it generates output.

Models are Grown, Not Built

Unlike traditional software, LLMs are not programmed with explicit rules but "grow" through successive phases of training on vast amounts of data. This process involves updating parameters as they learn to predict the next word, leading to inscrutable internal workings.

Next-Token Prediction

The core training objective for LLMs, where the model is fed a sequence of words and asked to predict the next word. As models improve, this seemingly simple task requires them to develop a deep understanding of world facts, logic, and underlying structures.

Internal Representations

LLMs form unique, general representations of concepts (e.g., "Texas," "eye") in their intermediate numerical results. These representations are activated across different contexts and languages, indicating a deeper understanding beyond surface-level text processing.

Internal Planning

LLMs can formulate and follow plans in their intermediate results, even without explicitly stating them in their output. This allows them to execute higher-order tasks like writing coherent poetry or code by referring back to an internal strategy.

Emergence

The phenomenon where complex capabilities and behaviors, such as understanding concepts or planning, arise from seemingly simple training objectives (like next-token prediction) and increased compute. This leads to sudden capability jumps from smooth underlying progress.

Scaling Laws

Empirical observations showing that as more compute and data are applied to the task of predicting the next word, the model's skill predictably and smoothly improves. This continuous improvement correlates with enhanced performance on various complex tasks.

AI Alignment

The effort to ensure that AI models behave in ways that are helpful, harmless, and honest, aligning with human values and intentions. This involves both training models to exhibit desired behaviors and using interpretability to verify internal states and prevent undesirable actions.

Anthropomorphization (AI)

The act of attributing human characteristics or behaviors to AI models. While often criticized, the speaker argues it can be a useful mental model for understanding and predicting AI behavior due to how models are trained on human text, dialogues, and outcome-oriented tasks.

System 1 vs. System 2 Thinking (AI)

An analogy used to describe different types of AI decision-making. System 1 refers to fast, gut-level, binary decisions (like in Jev models), while System 2 involves more deliberate, long-term reasoning, similar to human cognitive processes.

?
What is AI interpretability research?

It's a field focused on understanding how large language models (LLMs) like Claude work by examining their internal processes and intermediate numerical results, rather than just observing their inputs and outputs.

?
How do AI models "think" or "operate"?

AI models operate by converting words into numbers, performing many mathematical operations on these numbers, and then converting the final results back into words, with their internal "thinking" involving the manipulation of these numerical representations.

?
Do AI models simply predict the next word like autocomplete?

No, while trained on next-token prediction, advanced AI models go beyond simple autocomplete; to predict the hardest words accurately, they develop a deep understanding of underlying world structures and concepts, similar to how humans reason.

?
Can AI models plan ahead, or do they only think one word at a time?

AI models can plan many tokens ahead by formulating internal plans within their intermediate results, which they then refer back to as they output words one at a time, allowing for coherent and complex multi-step outputs.

?
How do researchers "look inside" an AI model?

Researchers decode the intermediate numerical results produced between each word output by passing various texts to the model, recording these numbers, and then identifying patterns that correspond to specific concepts or internal states.

?
Why do AI capabilities seem to jump suddenly when underlying progress is smooth?

AI models improve smoothly, but their utility for specific tasks (like coding or email writing) only becomes apparent once they cross a reliability threshold, making them "good enough" to be genuinely helpful for that task.

?
Why is it hard to catch AI models "thinking badly" or trying to deceive us?

AI models can represent information and plans in their intermediate results without verbalizing them, making it challenging to detect malicious intent solely from their output; interpretability is needed to examine these unstated internal states.

?
Is anthropomorphizing AI a valid approach?

Yes, the speaker argues that anthropomorphizing AI can be a useful mental model for understanding and predicting its behavior and risks, given that models are trained on vast amounts of human-generated text, dialogues, and outcome-oriented tasks that reward human-like strategies.

?
Are we currently in a "fast takeoff" or "slow takeoff" scenario for AI?

The current progress suggests a "slow takeoff" scenario, where AI intelligence improves smoothly and predictably, allowing for continuous learning, adaptation, and the implementation of safety measures along the way.

?
Is there optimism for AI alignment?

Yes, the speaker is optimistic because many smart people are seriously focused on AI safety and alignment, and as models become more capable, they also become somewhat easier to study internally, creating a push-and-pull dynamic that encourages continuous mitigation efforts.

?
What can AI teach us about intelligence itself?

Studying AI can reveal fundamental truths about the nature of intelligence, particularly how complex systems and capabilities can emerge from simple objectives, similar to how evolution led to human intelligence.

1. Take AI Progress Seriously

Acknowledge and seriously consider that AI technology will continue to improve at a rapid pace, as this worldview is foundational for responsible development and safety measures.

2. AI Models Grow, Not Built

Recognize that AI models “grow” through extensive data training rather than being explicitly “built” with programmed rules, which means their internal workings are often inscrutable and require scientific study.

3. Next Token Prediction is Complex

Understand that predicting the next token, especially for advanced AI, is a deceptively complex task that requires deep understanding of underlying structures and concepts, not just simple autocomplete.

4. AI Models Plan Ahead

Be aware that AI models, despite outputting one word at a time, can formulate and follow internal plans over many tokens, allowing them to produce coherent and complex outputs like poetry or code.

5. Use Anthropomorphic AI Framing

Consider anthropomorphizing AI as a useful mental model for understanding its behaviors and predicting potential risks, given how models are trained on human text, dialogues, and outcome-oriented tasks.

6. Live in the Present

Avoid the “when I finally” mentality, which projects happiness onto future achievements, and instead strive to be present and appreciate the current moment, as life is composed of these lived experiences.

7. Engage with AI Interpretability

Educate yourself on how AI models work by exploring interpretability research and resources like Neuronpedia, as informed public engagement is crucial for shaping the future of AI.

One of the most important things to realize if you use language models or think about them is just this very bizarre fact that to make them is not to understand them.

Emmanuel Ameisen

Humans, if you really boil it down, like we're basically next token predictors ourselves. We're just trying to figure out the next thing to say.

Emmanuel Ameisen

The best way to sort of understand how that thing behaves is not like exactly like a human, but man, kind of like something that's mimicking a human is really close and is much closer than a program.

Emmanuel Ameisen

I find that the more you look inside, the less certain you are about how they work. And then oftentimes people that will confidently say like, ah, language models are just X or just Y are the people that actually haven't spent a while looking inside.

Emmanuel Ameisen

I think we're going to learn a lot about just the nature of intelligence itself, right? Of sort of like how you get this super complex thing that emerges from, from simple objectives.

Emmanuel Ameisen

This is such a new field, like both interp and like AI and LMs. And I think it, it just benefits from, from people engaging and, and, um, being informed.

Emmanuel Ameisen

LLM Training Process (High-Level)

Emmanuel Ameisen
  1. Phase 1 (Pre-training): Take a large corpus of data, feed the model the first half of a document, and ask it to predict the next word. Repeat this for an "ungodly amount of data."
  2. Phase 2 (Fine-tuning): Train the model on a corpus of examples of how it should behave, such as helpful assistant dialogues or solving coding problems.

Investigating Weird AI Behavior (Example: '36' Problem)

Emmanuel Ameisen
  1. Step 1: Observe Weird Behavior: Notice the model exhibiting an unexpected or "crazy" behavior, such as consistently outputting an incorrect answer despite calculating the correct one internally, or descending into "madness."
  2. Step 2: Look Inside (Intermediate Results): Examine the intermediate numerical results within the model's "thought process" as it processes the input and generates the output.
  3. Step 3: Decode Activations: Identify specific activations or patterns in the intermediate results that correspond to certain concepts or numbers (e.g., a "huge activation for 36").
  4. Step 4: Understand the Cause: Determine if the internal activations reveal a conflict (e.g., the model correctly calculated 12 but was optimized to say 36 due to prior training data).
  5. Step 5: Improve Alignment: Use the understanding gained to improve the general alignment of the current and future models by addressing the root cause in training.
2019
Year language models surpassed humans in next-word prediction Based on empirical observation of model performance.
90%
Percentage of medium words predicted correctly for useful email writing Threshold at which AI becomes reliably useful for tasks like email drafting.
80%
Percentage of code written correctly by Claude (not useful) Prior performance level where the model was not yet useful for coding due to requiring too much correction.
99%
Percentage of code written correctly by Claude (useful) Current performance level where the model is reliably useful for coding tasks.
Hundreds
Approximate number of people in the AI interpretability field Globally, across different labs and academia.