What do we know for sure about human psychology? (with Simine Vazire)
1. Pre-register Research Plans
To improve replicability, pre-register a detailed research plan, committing to follow it as closely as possible, and if deviations occur, document them and re-test the revised plan.
2. Conduct Direct Replications
After refining a research design, conduct direct replications by repeating the study exactly the same way to confirm the robustness of findings, especially when combined with pre-registration.
3. Verify Pre-registration Details
When reviewing a paper, actively check the pre-registration document, as authors may misrepresent the extent of their planning, and a vague or deviated plan can indicate issues.
4. Be Skeptical of Flexible Research
Maintain skepticism towards research findings that show hallmarks of flexibility, such as significant deviations from a pre-registered plan, vague initial plans, or results barely meeting statistical significance.
5. Demand Research Materials
As a reviewer, always request and examine the research materials (e.g., surveys, stimuli) and refuse to review papers where they are not provided, as this is crucial for understanding the study’s execution.
6. Reject Oversold Research
Journal editors should reject papers that significantly oversell their findings, even if the issues are fixable, to counteract perverse incentives for authors to exaggerate their work.
7. Prioritize Simplest Valid Analysis
When analyzing data, always perform the simplest valid analysis first to establish a baseline and compare it with more complex methods, as fancy analyses can sometimes obscure or misrepresent findings.
8. Shift Academic Incentive Focus
Change academic incentives to be skeptical of researchers who routinely claim breakthrough discoveries, instead valuing careful, robust work over frequent, flashy findings, especially in complex fields like psychology.
9. Triangulate Research Methods
To improve research validity, triangulate data using multiple methods (e.g., self-reports, peer reports, archival records, physiological measures), ensuring the predictor and outcome are not measured using the same method.
10. Be Skeptical of Self-Reports
Treat self-report measures with skepticism, as self-concept, interpretation, and recall problems can significantly influence answers, making them rarely objective measures of psychological constructs.
11. Heed P-Value Strength
Be more confident in findings with P-values of 0.01 or smaller, as empirical data suggests these results are significantly more likely to replicate than those with larger P-values.
12. Read Methods Before Story
When reading research papers, start by reviewing the methods and results, and check the materials, before engaging with the authors’ narrative, to avoid being swayed by storytelling and better assess the actual findings.
13. Prioritize Therapeutic Connection
When choosing a therapist, prioritize finding someone with whom you have a good connection, as the therapeutic relationship matters a lot and a poor connection can signal that things aren’t going well.
14. Value Descriptive Research
Prioritize descriptive research to understand ‘what’ is happening before delving into ‘why,’ as robust effects can serve as foundational building blocks for later theoretical development.
15. Make Circumscribed Research Claims
When research designs don’t support strong causal inferences, make more circumscribed claims, such as descriptive observational claims, to build foundational knowledge for future causal theories.
16. Rethink Relationship Success Metrics
Re-evaluate the definition of positive outcomes in relationships research, as staying together is not always the goal; consider the quality of the relationship over its duration, and acknowledge that breaking up can be a positive outcome.
17. Expand Relationships Research Focus
Broaden the scope of relationships research beyond romantic relationships to include other important connections like friendships, as there is a significant lack of understanding in these areas.
18. Consider Narrower Personality Traits
When analyzing personality, consider using more fine-grained facets or alternative frameworks like narratives and motives, as narrower trait definitions can increase predictive accuracy.
19. Avoid Gratuitous Physiological Measures
Do not assume non-self-report measures, like physiological data, are inherently superior; ensure they genuinely align with the construct being measured, as sometimes self-reports can be more appropriate.
20. Automate Non-Judgmental Peer Review
Utilize generative AI or simple algorithms to automate non-judgmental aspects of peer review, freeing up human labor for critical thinking and judgment that machines cannot yet replicate.
21. Set Realistic Peer Review Expectations
Understand that pre-publication peer review is crucial but has limitations, performing only about 10% of what people assume; it’s not a comprehensive solution for ensuring research quality.
22. Be Wary of Silver Bullet Tools
Be skeptical of claims that new statistical tools like Bayesian methods are a ‘silver bullet’ for research problems, as the core issue often lies with human factors and misuse rather than the tools themselves.