Scientific Progress and the Replication Crisis (with Geoff Anders)

1. Adopt an Engineering Mindset

Approach scientific fields with an engineering mindset, focusing on practical outcomes and refining measurement tools, rather than abstract hypothesis testing on poorly understood systems. This helps build reliable phenomena and instruments, as seen in successful historical sciences.

2. Prioritize High-Volume, Low-Cost Replication

Develop and utilize research methods that allow for extremely low-cost and high-volume replication of observations. This enables continuous instrument refinement and builds a robust, reliable knowledge base, crucial for fields like psychology.

3. Embrace Iterative Study Design

Adopt an iterative approach to designing and conducting studies, recognizing that initial experiments may primarily serve to refine methodology and understanding. This allows for learning how to conduct effective studies through repeated attempts, rather than expecting perfect results from a single trial.

4. Refine Introspection as an Instrument

Treat introspection as a skill or instrument that can be refined and improved for more reliable self-understanding. Actively work to understand its reliability and limitations, similar to how scientific tools are developed and calibrated.

5. Ask ‘Why’ in Surveys

When conducting surveys or research, ask participants to explain their answers to uncover underlying motivations and refine measurement instruments. This helps identify if questions are truly measuring what they intend, as demonstrated by the sunk cost fallacy example.

6. Practice the ‘Inner Why’ Technique

To better understand your emotions, practice observing and identifying the precise moment an emotion shifts. This makes it easier to pinpoint the immediate cause, as usually little else has changed in those few seconds.

7. Interpret Emotions as Information

Recognize that different emotions carry distinct kinds of information about your interpretation of events. For example, anger might signal a perceived block or loss of value, while sadness might indicate an actual loss, guiding deeper self-exploration.

8. Note Thoughts During Emotional Shifts

When you experience an emotional shift, immediately pay attention to the thoughts you were having or the event that just occurred. This helps connect specific thoughts and external triggers to your emotional responses, a core principle of cognitive behavioral therapy.

9. Observe Spontaneous Thoughts

Practice mindful observation to notice that thoughts often appear spontaneously in your awareness, rather than always being consciously controlled. This can be a powerful insight gained through careful introspection, as taught in some meditation schools.

10. Identify Knowledge Gaps from Practical Skills

Look for domains where people reliably perform complex tasks (e.g., leadership, conflict resolution) but where scientific theories don’t yet explain how they do it. These areas indicate significant untapped knowledge for scientific discovery.

11. Foster Collaborative Checking of Discoveries

Create and support environments that encourage easy sharing and rigorous checking of scientific discoveries among peers. This makes it harder for individuals to ‘game’ the system and enhances the overall integrity and reliability of collective knowledge.

12. Utilize the Focusing Technique

Learn and practice Eugene Gendlin’s Focusing technique, which involves finding ‘felt senses’ in your body and attending to them to find words that capture what’s going on. This introspective method can help individuals better understand their internal experiences.

13. Challenge Preconceived Self-Concepts

Actively challenge your self-concept by observing your actual emotional reactions to events, even if they contradict how you believe you ‘should’ react. This helps uncover genuine patterns of response and leads to more accurate self-understanding.