What do we know for sure about human psychology? (with Simine Vazire)
Spencer Greenberg and Sameen Vazir discuss the state of psychological research, its slow progress on important topics, and the need for a culture of scrutiny and transparency to address issues like the replication crisis and improve the field's reliability.
Deep Dive Analysis
17 Topic Outline
Psychology's Progress and Challenges on Important Topics
Success Stories in Psychology: Psychophysics and Clinical Therapy
Debates and Nuances of the Big Five Personality Model
Challenges of Self-Report Measures in Psychological Research
Social Psychology's Potential for Progress: Prejudice and Relationships
Replicability of Classic Social Psychology Findings
The Replication Crisis and the Need for Transparency
Moving Beyond Transparency: The Purpose of Open Science
The 'Replication Crisis of Everything' in Psychology
Evidence of Improved Replicability in Recent Research
Strategies to Improve Replicability: Pre-registration and Skepticism
The Impact of Sample Size and Researcher Behavior on False Positives
The Problem of Fraud in Psychological Research
The Issue of 'Importance Hacking' and Perverse Publication Incentives
Rapid-Fire: Bayesian Stats, Peer Review, and Academic Incentives
Role of Technology and AI in Improving Psychological Science
Drawbacks of Democracy and the Role of Constitutions
9 Key Concepts
Psychophysics
A field of psychology that studies how people process low-level sensory information, often through experiments like flashing lines on a screen. It is characterized by a slow and steady approach to scientific progress.
Therapeutic Alliance
The quality of the relationship and connection between a therapist and their client. It is considered an important factor in the effectiveness of therapy, though the direction of causality (whether it causes improvement or is a sign of it) is still debated.
Big Five Personality Model (OCEAN)
A widely used framework in personality psychology that identifies five core dimensions of personality: Openness, Conscientiousness, Extroversion, Agreeableness, and Neuroticism. It is derived from statistical analysis of self-reported personality attributes.
Shared Method Variance
An artificial correlation between two self-report measures that arises solely because they use the same measurement method, such as a tendency for respondents to consistently circle high or low numbers, rather than reflecting a true underlying relationship between the constructs.
Replicability
The ability to consistently reproduce the results of a study when the experiment is conducted again, ideally under similar conditions. A lack of replicability has led to a 'replication crisis' in psychology, questioning the reliability of past findings.
Generalizability
The extent to which research findings can be applied or extended to different populations, settings, or measures beyond the specific conditions of the original study. It addresses whether a finding is robust across various contexts, not just whether it can be reproduced.
Pre-registration
The practice of publicly documenting a detailed research plan, including hypotheses, methods, and analysis strategy, before collecting or analyzing data. This helps prevent p-hacking and selective reporting, ensuring that researchers stick to their initial intentions.
False Discovery Rate
The proportion of statistically significant findings that are actually false positives among all significant results. It is influenced by factors like statistical power, where lower power (e.g., due to small sample sizes) can lead to a higher false discovery rate.
Importance Hacking
The act of overselling or misrepresenting the significance, novelty, or implications of research findings to enhance their perceived value and increase publication chances. This can occur even when the actual empirical contribution is minor, trivial, or based on methodological artifacts.
10 Questions Answered
Simine Vazire is pessimistic, suggesting that the importance and interest in these topics make it harder to conduct slow, careful research, leading to less progress than desired.
Psychophysics, which studies low-level information processing, is cited as an area of steady progress. In clinical psychology, the effectiveness of therapy and the importance of the therapeutic relationship are considered significant findings.
While widely used for standardization, there's ongoing debate within personality psychology about the exact number of dimensions (e.g., Hexaco's sixth factor), whether traits are the best unit of analysis, and the level of granularity (facets vs. broad traits).
Simine Vazire holds an extreme view that few things are objective enough to be immune to self-concept influencing answers, even without intentional lying, due to definitional ambiguity, recall problems, and interpretation.
Prejudice, especially through interdisciplinary field studies, and relationships (beyond just romantic ones, including friendships) are highlighted as areas with significant potential for more rigorous research.
There is cautious optimism and some early evidence, such as from the Transparent Replications project, suggesting that replicability in top journals may have substantially improved in the last decade, possibly by as much as double.
Simine Vazire, based on trusted sources, estimates that the percentage of papers containing fraud could be closer to 10%, which is higher than commonly assumed.
Around 15 years ago, it's estimated that about 40% of papers in top journals would not replicate. This rate is believed to have improved substantially, possibly to around 20% or less now.
Key steps include pre-registering detailed research plans, conducting direct replications, and maintaining skepticism towards findings that show hallmarks of flexibility, such as vague plans or results barely meeting significance thresholds.
Simine Vazire argues that setting too high a bar for publication can create perverse incentives for 'importance hacking' (overselling findings), suggesting that journals should accept incremental, solid, even 'boring' research to foster better scientific practices.
22 Actionable Insights
1. Pre-register Research Plans
To improve replicability, pre-register a detailed research plan, committing to follow it as closely as possible, and if deviations occur, document them and re-test the revised plan.
2. Conduct Direct Replications
After refining a research design, conduct direct replications by repeating the study exactly the same way to confirm the robustness of findings, especially when combined with pre-registration.
3. Verify Pre-registration Details
When reviewing a paper, actively check the pre-registration document, as authors may misrepresent the extent of their planning, and a vague or deviated plan can indicate issues.
4. Be Skeptical of Flexible Research
Maintain skepticism towards research findings that show hallmarks of flexibility, such as significant deviations from a pre-registered plan, vague initial plans, or results barely meeting statistical significance.
5. Demand Research Materials
As a reviewer, always request and examine the research materials (e.g., surveys, stimuli) and refuse to review papers where they are not provided, as this is crucial for understanding the study’s execution.
6. Reject Oversold Research
Journal editors should reject papers that significantly oversell their findings, even if the issues are fixable, to counteract perverse incentives for authors to exaggerate their work.
7. Prioritize Simplest Valid Analysis
When analyzing data, always perform the simplest valid analysis first to establish a baseline and compare it with more complex methods, as fancy analyses can sometimes obscure or misrepresent findings.
8. Shift Academic Incentive Focus
Change academic incentives to be skeptical of researchers who routinely claim breakthrough discoveries, instead valuing careful, robust work over frequent, flashy findings, especially in complex fields like psychology.
9. Triangulate Research Methods
To improve research validity, triangulate data using multiple methods (e.g., self-reports, peer reports, archival records, physiological measures), ensuring the predictor and outcome are not measured using the same method.
10. Be Skeptical of Self-Reports
Treat self-report measures with skepticism, as self-concept, interpretation, and recall problems can significantly influence answers, making them rarely objective measures of psychological constructs.
11. Heed P-Value Strength
Be more confident in findings with P-values of 0.01 or smaller, as empirical data suggests these results are significantly more likely to replicate than those with larger P-values.
12. Read Methods Before Story
When reading research papers, start by reviewing the methods and results, and check the materials, before engaging with the authors’ narrative, to avoid being swayed by storytelling and better assess the actual findings.
13. Prioritize Therapeutic Connection
When choosing a therapist, prioritize finding someone with whom you have a good connection, as the therapeutic relationship matters a lot and a poor connection can signal that things aren’t going well.
14. Value Descriptive Research
Prioritize descriptive research to understand ‘what’ is happening before delving into ‘why,’ as robust effects can serve as foundational building blocks for later theoretical development.
15. Make Circumscribed Research Claims
When research designs don’t support strong causal inferences, make more circumscribed claims, such as descriptive observational claims, to build foundational knowledge for future causal theories.
16. Rethink Relationship Success Metrics
Re-evaluate the definition of positive outcomes in relationships research, as staying together is not always the goal; consider the quality of the relationship over its duration, and acknowledge that breaking up can be a positive outcome.
17. Expand Relationships Research Focus
Broaden the scope of relationships research beyond romantic relationships to include other important connections like friendships, as there is a significant lack of understanding in these areas.
18. Consider Narrower Personality Traits
When analyzing personality, consider using more fine-grained facets or alternative frameworks like narratives and motives, as narrower trait definitions can increase predictive accuracy.
19. Avoid Gratuitous Physiological Measures
Do not assume non-self-report measures, like physiological data, are inherently superior; ensure they genuinely align with the construct being measured, as sometimes self-reports can be more appropriate.
20. Automate Non-Judgmental Peer Review
Utilize generative AI or simple algorithms to automate non-judgmental aspects of peer review, freeing up human labor for critical thinking and judgment that machines cannot yet replicate.
21. Set Realistic Peer Review Expectations
Understand that pre-publication peer review is crucial but has limitations, performing only about 10% of what people assume; it’s not a comprehensive solution for ensuring research quality.
22. Be Wary of Silver Bullet Tools
Be skeptical of claims that new statistical tools like Bayesian methods are a ‘silver bullet’ for research problems, as the core issue often lies with human factors and misuse rather than the tools themselves.
5 Key Quotes
I'm pretty pessimistic about the amount of progress we've made so far, but I think that the positive and the negative go together, right? I think it's partly because these topics are so important and so interesting. I think that makes it harder to do things the slow and careful way.
Simine Vazire
I think one really big challenge of social psych and psych in general is that there are probably super complicated interaction effects all over the place, but they're like, eight way interactions, they're not two or three way interactions. And they're not, we're never going to be able to detect them empirically, I think, even with all the data in the world.
Simine Vazire
The problem is not so much the peer review, but that it can't possibly be enough. We can't stop at just pre-publication peer review. That's not nearly enough.
Simine Vazire
I feel like I was going back to my like, undergrad level, first year grad student level view of science. And it had been beaten out of me with like all these, yeah, like you have to be strategic, blah, blah, blah, not consciously or intentionally. But I think what I see in the next generation is a little bit more of like holding on to those ideals.
Simine Vazire
I think it really makes psychology look bad that we expect and celebrate that people have these kinds of breakthrough level discoveries routinely. I think we're kind of addicted to that feeling and we need to stop.
Simine Vazire
1 Protocols
Recipe for Improving Research Replicability
Simine Vazire- Plan everything ahead of time and follow a detailed plan (pre-register).
- If the initial plan isn't right, tweak it, then conduct the study again exactly the same way (direct replication) once the recipe is down.
- Be skeptical of findings that have hallmarks of flexibility, such as deviations from vague plans or results that are barely statistically significant.