Scientific Progress and the Replication Crisis (with Geoff Anders)
Spencer Greenberg and Jeff Anders discuss the rate of scientific progress, the replication crisis, and their differing approaches to social science. They explore the need for an "engineering mindset" in research, emphasizing instrument refinement, rapid iterative experimentation, and the development of introspection as a reliable scientific tool.
Deep Dive Analysis
12 Topic Outline
Assessing the Speed and Nature of Scientific Progress
AI Progress: Compute, Algorithms, and GPT-3
Theories on Scientific Slowdown: Low-Hanging Fruit vs. Institutions
The Potential for Discovery in Social Sciences
Engineering Mindset vs. Abstract Hypothesis Testing in Science
Galileo's Telescope and the Process of Instrument Refinement
Challenges in Psychological Measurement: The Big Five and Sunk Cost Fallacy
The Replication Crisis in Social Science and its Implications
Introspection as a Scientific Tool for Psychological Understanding
Iterative, Fast-Paced Social Science Research
Impact of Academia's Expansion on Scientific Standards and Accumulation
Gameability of Scientific Systems and Incentives for Truth
7 Key Concepts
GPT-3
GPT-3 is an AI model trained on massive amounts of text data, whose sole function is to predict what text is likely to come next. This single task encompasses many different intelligence tasks, such as solving analogies, writing poetry, generating code, and simple language translation, due to the sheer scale of its training data and parameters.
Sunk Cost Fallacy
The sunk cost fallacy is the tendency for people to continue investing effort, time, or money into something, even past the point where it's a good investment, because they don't want their previous efforts to be wasted. Measuring this accurately in surveys requires careful question design to avoid confounding factors like politeness or guilt.
Instrument Refinement
Instrument refinement is the iterative process of improving measurement tools to make them more reliable and accurate. This involves repeatedly testing the instrument, identifying its flaws or unintended measurements, and modifying it based on those observations, similar to how early astronomers improved telescopes or how survey designers refine questions.
Replication Crisis
The replication crisis refers to the phenomenon in social science where many published studies, when attempted to be reproduced by other researchers, fail to yield the original findings. This raises concerns about the reliability of published research and the foundations upon which new knowledge is built.
Engineering Mindset (in Science)
An engineering mindset in science focuses on accomplishing a particular practical goal and extracting theory from that process, rather than starting with an abstract hypothesis and testing it on a poorly understood system. This approach emphasizes developing reliable phenomena and measurement tools through repeated, high-volume testing and refinement.
Felt Senses (Focusing)
Felt senses are bodily sensations or impressions that arise in response to a situation or problem, which are then attended to and explored to find words that accurately capture their meaning. This concept is central to Eugene Gendlin's Focusing technique, an introspective method for self-understanding.
Gameability of Systems
Gameability refers to how easily a system can be manipulated or exploited by individuals to achieve personal gain, often at the expense of the system's intended goals. In academia, reliance on legible metrics like publication count or journal impact factor can lead to gaming, where researchers prioritize these metrics over genuine scientific discovery.
7 Questions Answered
Assessing the current rate of scientific progress is extremely difficult because metrics like the number of papers published are rising astronomically, yet there's a sense that actual breakthroughs are not increasing at the same pace. Different fields progress at different rates, and what constitutes 'progress' itself (e.g., algorithmic advances vs. increased compute) is debatable.
Two broad narratives exist: the 'low hanging fruit' idea, suggesting that easier discoveries have been made and new ones are inherently harder, and the 'human culture or institutions' idea, which posits that scientific processes are becoming less efficient or more bureaucratic, hindering progress.
It's challenging because survey questions can inadvertently measure other phenomena, such as politeness or guilt, rather than the pure sunk cost effect. Researchers must carefully refine questions through iterative testing and qualitative feedback (like asking 'why') to isolate the intended psychological construct.
The replication crisis is the observation that many published social science findings do not hold up when other researchers attempt to replicate them. This makes it difficult to discern which findings are reliable, hindering the accumulation of a stable body of scientific knowledge and making researchers reluctant to build on previous work.
Introspection can be a powerful, low-cost tool for psychological research, allowing individuals to observe their internal states and processes. However, it requires refinement, similar to a scientific instrument, by developing clear instructions and understanding individual differences in interpretation, to ensure reliable and transmittable observations.
As academia expands, the increased number of researchers makes personal judgment of colleagues difficult, leading to a push for more 'legible' standards like publication count or journal impact factor. This can shift focus away from genuine discovery, and while discoveries are still made, they may not be effectively accumulated into a standing body of knowledge due to issues like the replication crisis.
One technique is to pay close attention to the exact moment an emotion shifts, as this narrow timeframe makes it easier to identify the specific thought or event that triggered the change. Additionally, recognizing the different information types conveyed by various emotions (e.g., anger often signals a perceived block, sadness a loss) can provide fruitful lines of inquiry.
13 Actionable Insights
1. Adopt an Engineering Mindset
Approach scientific fields with an engineering mindset, focusing on practical outcomes and refining measurement tools, rather than abstract hypothesis testing on poorly understood systems. This helps build reliable phenomena and instruments, as seen in successful historical sciences.
2. Prioritize High-Volume, Low-Cost Replication
Develop and utilize research methods that allow for extremely low-cost and high-volume replication of observations. This enables continuous instrument refinement and builds a robust, reliable knowledge base, crucial for fields like psychology.
3. Embrace Iterative Study Design
Adopt an iterative approach to designing and conducting studies, recognizing that initial experiments may primarily serve to refine methodology and understanding. This allows for learning how to conduct effective studies through repeated attempts, rather than expecting perfect results from a single trial.
4. Refine Introspection as an Instrument
Treat introspection as a skill or instrument that can be refined and improved for more reliable self-understanding. Actively work to understand its reliability and limitations, similar to how scientific tools are developed and calibrated.
5. Ask ‘Why’ in Surveys
When conducting surveys or research, ask participants to explain their answers to uncover underlying motivations and refine measurement instruments. This helps identify if questions are truly measuring what they intend, as demonstrated by the sunk cost fallacy example.
6. Practice the ‘Inner Why’ Technique
To better understand your emotions, practice observing and identifying the precise moment an emotion shifts. This makes it easier to pinpoint the immediate cause, as usually little else has changed in those few seconds.
7. Interpret Emotions as Information
Recognize that different emotions carry distinct kinds of information about your interpretation of events. For example, anger might signal a perceived block or loss of value, while sadness might indicate an actual loss, guiding deeper self-exploration.
8. Note Thoughts During Emotional Shifts
When you experience an emotional shift, immediately pay attention to the thoughts you were having or the event that just occurred. This helps connect specific thoughts and external triggers to your emotional responses, a core principle of cognitive behavioral therapy.
9. Observe Spontaneous Thoughts
Practice mindful observation to notice that thoughts often appear spontaneously in your awareness, rather than always being consciously controlled. This can be a powerful insight gained through careful introspection, as taught in some meditation schools.
10. Identify Knowledge Gaps from Practical Skills
Look for domains where people reliably perform complex tasks (e.g., leadership, conflict resolution) but where scientific theories don’t yet explain how they do it. These areas indicate significant untapped knowledge for scientific discovery.
11. Foster Collaborative Checking of Discoveries
Create and support environments that encourage easy sharing and rigorous checking of scientific discoveries among peers. This makes it harder for individuals to ‘game’ the system and enhances the overall integrity and reliability of collective knowledge.
12. Utilize the Focusing Technique
Learn and practice Eugene Gendlin’s Focusing technique, which involves finding ‘felt senses’ in your body and attending to them to find words that capture what’s going on. This introspective method can help individuals better understand their internal experiences.
13. Challenge Preconceived Self-Concepts
Actively challenge your self-concept by observing your actual emotional reactions to events, even if they contradict how you believe you ‘should’ react. This helps uncover genuine patterns of response and leads to more accurate self-understanding.
6 Key Quotes
I think that there's an important difference between the question of whether there will be a simple unified theory versus whether the thing can be understood at all.
Geoff Anders
I'd say I'm worried the standards are low, right? Whereas when you're engineering, when you're building something, when you need people to have a particular experience, when you're trying to make a company that operates efficiently, the company itself is a different example, is composed of tons of different parts. There's implicit parts, explicit, you've got the people, the culture, the things they're working with, internal, external audiences, it's all super complicated. And it has to be designed by people like, you know, businesses don't assemble themselves.
Geoff Anders
It's like so much of that doesn't hold up. How are you going to build on top of that?
Spencer Greenberg
a paper is not the science itself. It's like, you know, a marketing version of the science, right?
Spencer Greenberg
I think that overall, there's... I mean, of course, there will be people who will unscrupulously game. And then, of course, there will be tons of sort of subconscious or unconscious gaming where the people need to publish in order to advance. Like, absolutely. But I still think that there are just tons and tons of people inside academia and outside of the people who are doing research and trying to figure things out who are genuinely motivated to find the truth.
Geoff Anders
If you figure out something important, then once the field eventually figures that out, then they remember it.
Geoff Anders
2 Protocols
Focusing Technique
Geoff Anders- Find felt senses in your body.
- Attend to these felt senses.
- Try to find words that capture what's going on with them.
Inner Why Technique
Spencer Greenberg- Learn to notice the exact moment your emotion shifts.
- Identify what specific thought or event immediately preceded the emotional change.
- Reflect on the type of information the emotion conveys (e.g., anger for perceived blocking, sadness for loss) to explore its meaning.