How f***ed is psychology? (with Adam Mastroianni)
Adam Mastroianni, a psychologist and metascientist, critiques the current state of psychology research, discussing the replication crisis, the pitfalls of peer review, and the importance of science communication. He advocates for more honest, accessible, and theoretically grounded scientific approaches to advance understanding.
Deep Dive Analysis
15 Topic Outline
The Current State and Future Hope for Psychology
Psychology's Delayed Start Compared to Physics
The Role of Intuition and Cognitive Biases in Psychology
Critique of Social Science Mega-Studies
Re-analysis of a Gym Attendance Mega-Study
Effective Interventions for Habit Formation
The Nature of Scientific Progress: Entities and Rules
Spencer's Theory of Depression and Anxiety
Misunderstandings from the Replication Crisis
The 'So What?' Test for Psychology Research
Adam's 'Stupidest Study' and Learning from Simple Experiments
The Value of Honest and Accessible Science Communication
Pros and Cons of Academic Peer Review
Transparent Replications Project and its Impact
Recommendations for Advancing Psychology Research
8 Key Concepts
Folk Psychology
This refers to the natural, everyday understanding of human behavior and mental states that people develop through experience. While often accurate enough for daily survival, it sets a high bar for scientific psychology to surpass or correct.
Mega-Studies
These are large-scale research projects that test numerous interventions simultaneously against each other or a control group. The goal is to identify which interventions are most effective, often in a specific domain like habit formation.
Importance Hacking
This is a phenomenon where authors strategically frame their research findings in abstract or grand terms to make them appear more valuable or worthy of publication than the literal, often mundane, actions or outcomes of the study.
Science as Entities and Rules
This philosophical view suggests that scientific progress fundamentally involves proposing what the world is made of (entities) and the principles governing their interactions (rules), rather than merely testing isolated empirical facts.
Depression (Spencer's Theory)
Proposed as a brain state resulting from the prediction that one cannot produce anything of value. This leads to symptoms like excessive sleep, tiredness, and hopelessness, and is often highly correlated with anxiety.
Anxiety (Spencer's Theory)
Proposed as a prediction that things one values are likely to be destroyed. It is distinct from depression but highly correlated due to causal loops and shared underlying causes.
Value (Spencer's Theory)
Described as a fundamental mental operation where the brain evaluates something as 'good' or 'bad,' whether it's a real-world object, a hypothetical situation, or a memory. This evaluation drives behavior.
Replication Crisis
A period in social science where many previously published findings, particularly in psychology, failed to produce the same results when re-tested. This highlighted issues with research methods and the need for greater rigor.
6 Questions Answered
Psychology's formal start in the 1860s, compared to physics in the late 1500s/early 1600s, is likely due to the strength and satisfying nature of human intuition about psychology, making it harder to realize when our understanding is incomplete or wrong, unlike in physics where intuitions break down more easily in novel situations.
Mega-studies can identify interventions that work in specific contexts, but they often lack theoretical grounding, making it difficult to understand *why* an intervention works or if it will generalize to other situations or populations. They can be inefficient for building fundamental understanding of the mind.
A mega-study on gym attendance found that giving people bonuses for returning after missed workouts, offering higher incentives, and allowing people to choose between gain or loss framing for rewards seemed to increase habit adherence, though the effects were generally moderate.
Beyond improving methodological rigor (e.g., sufficient power, avoiding false positives), the primary lesson should be to prioritize research that would *matter* if true, rather than focusing solely on whether a finding is statistically significant or replicable, especially when the initial setup is artificial or abstract.
Blogging allows for more honest, engaging, and accessible writing, reaching a broader audience and often eliciting higher quality, more constructive feedback from motivated readers. It also avoids the lengthy, adversarial, and often unhelpful peer review process, and makes research freely available.
Peer review is often painful, time-consuming, and inefficient, with reviewers sometimes providing minor or unhelpful suggestions, or even forcing researchers to adopt invalid practices. It can also create an adversarial relationship, incentivize dishonesty, and destroy valuable information by not publishing reviews or rejected studies.
18 Actionable Insights
1. Prioritize Research by Impact
Before conducting any research, ask if the finding would truly matter if it were proven true. If the answer is no, then the effort might be better spent on questions with higher stakes, regardless of their truth value.
2. Define Science by Entities & Rules
Approach scientific inquiry by proposing the fundamental entities that constitute the world and the rules governing their interactions. This method, successful in other disciplines, helps build robust models rather than just testing abstractions.
3. Embrace Variance as Information
Instead of dismissing individual differences or ’noise’ in data, view variance as valuable information. Understanding why certain things work for some people or situations but not others can lead to deeper insights into underlying mechanisms.
4. Observe Natural Behavior First
Before designing artificial experiments, focus on explaining the vast amount of free behavioral data available from people’s everyday lives. This raw material can provide foundational insights without the need for costly, controlled studies.
5. Prioritize Truth-Seeking Over Appearance
In research, prioritize genuinely seeking the truth, even if it means deviating from conventional ‘scientific-looking’ practices or uniform criteria. Be transparent about subjective decisions and the reasoning behind them.
6. Write Science with Honesty & Clarity
Communicate scientific findings directly, honestly, and in a normal, engaging voice, avoiding abstractions and dissembling. This makes research more accessible to a broader audience, including non-experts, and fosters better understanding.
7. Practice Full Research Transparency
Share all raw materials, exact questions asked, and even studies that didn’t fit the final narrative, as digital space is essentially free. This provides valuable context for others, allows for deeper scrutiny, and facilitates future research.
8. Seek Feedback from Truth-Seekers
Actively solicit feedback from individuals genuinely interested in improving the work, rather than relying solely on adversarial peer review processes. This often yields higher quality and more constructive input, fostering collaboration.
9. Rapidly Correct Research Errors
When errors are identified in published work, promptly acknowledge and correct them, even if the work is already public. This demonstrates a commitment to accuracy and fosters a culture of continuous improvement in science.
10. Use Intuition Violations for Discovery
Identify informative facts by looking for instances where human behavior deviates from rational expectations or strong intuitive predictions. These ‘surprising’ findings often point to deeper underlying mechanisms and are valuable for scientific progress.
11. Conduct Micro-Experiments with Predictions
Run small, rapid experiments (e.g., Twitter polls) with pre-registered predictions to quickly test hypotheses. Learning occurs most effectively when results drastically deviate from expectations, prompting re-evaluation of assumptions.
12. Engage in Philosophical Reflection
Regularly reflect on the history and philosophy of science to understand the fundamental purpose and methods of psychological inquiry. This helps clarify what one is truly trying to achieve in research and guides methodological choices.
13. Foster Diverse & ‘Weird’ Research
Encourage and support unconventional research approaches and institutions that explore ideas outside the mainstream. This diversity is crucial for generating novel insights and pushing scientific boundaries, as good ideas don’t always look ’normal’.
14. Implement Daily Ritual for Habits
Utilize a structured habit formation tool, such as the ‘Daily Ritual’ described, which combines multiple techniques identified through randomized trials. This approach has been shown to increase habit adherence more effectively.
15. Offer Bonuses for Missed Workout Recovery
To improve habit adherence, provide extra incentives or bonuses for individuals who return to a habit after a missed session. This strategy was found to be effective in increasing gym attendance in a mega-study.
16. Allow Choice in Gain/Loss Framing
When providing incentives for habits, offer individuals the choice to frame their progress as either gaining rewards or avoiding losses. This personal choice was found to improve gym retention in a mega-study.
17. Increase Incentives for Desired Behavior
Recognize that higher incentives generally lead to more frequent engagement in desired behaviors. Paying people more to do things tends to increase their participation, a consistently observed phenomenon.
18. Anticipate Emotional Adaptation
Be aware that people often overestimate the duration of their emotional reactions to events. Expect to return to a baseline emotional state over time, as normal routines and daily life tend to restore normalcy.
7 Key Quotes
I think in the same way that like, we're in our prehistory. And, you know, just like you could have been, you know, looking at the field of alchemy in the middle of the 1500s, and gone like, I don't know, man, it doesn't look like it's really going anywhere. But eventually it did. Like, I think we're in that same period.
Adam Mastroianni
The real valuable knowledge is the knowledge that we understand so well that we know whether it's going to work or not because we have the model down.
Adam Mastroianni
The thing that we need to do with every new piece of research that comes out is first ask, is it true? When in fact, I think the first thing we need to ask is, would it matter if it were true? And if the answer to that question is no, then you don't have to care whether it's true or not.
Adam Mastroianni
I really wish that every psychology paper just had a table at the top that was like, here's exactly what we actually did. Like, here's literally what we had participants do.
Spencer Greenberg
I was sick of writing the bad version of the paper. I was, like, sick of lying and falling over myself to explain how this, like, so clearly, you know, fit in with previous research when it was like, no, this just totally came out of us just, like, pulling something out of our ass and then thinking that we knew what was going on and then not knowing what was going on.
Adam Mastroianni
I'm not a nudist, never been to a nudist thing, but I imagine this is what it feels like to go to a retreat where everyone's naked, that you're just like, now, like I am what I am. And like, I can look good. I can look bad, but this is how I look. And I don't have to worry anymore about how I look. It's so freeing.
Adam Mastroianni
If your work is good and you keep doing it, it just seems extremely likely that it's going to find its way to the people who are interested in it.
Adam Mastroianni