Statistics Intuitions and Social Science Reproducibility (with Stuart Buck)

Jul 7, 2021 1h 13m 12 insights Episode Page ↗
Spencer Greenberg and Stuart Buck discuss improving math education by focusing on statistics and logic, the nuances of p-values and publishing null results, and the critical need for open science, reproducibility, and better research practices in various fields.
Actionable Insights

1. Integrate Continuous RCTs

Weave lightweight randomized control trials (RCTs) into the deployment of interventions (e.g., cash transfers, digital apps) to continuously collect high-quality data on the target population and iterate for improvement, rather than relying on generalization from other studies.

2. Question Your Successes

Adopt the practice of questioning your successes as much as your failures, like good poker players, to determine if your decision-making process was sound or if you merely got lucky.

3. Prioritize Core Math & Logic

Restructure high school math curriculum to prioritize basic statistics, data analysis, and direct logic instruction, as these concepts are more useful for understanding the world, news, and scientific claims than esoteric geometry or indirect logic training.

4. Master Core Statistical Concepts

Learn and understand fundamental statistical concepts like the mean, median, probability distributions (e.g., standard deviation), and the difference between correlation and causation, as these are critical for making sense of the world and scientific information.

5. P-values for Sampling Error

Use p-values as a tool to reasonably rule out sampling error, understanding that a very low p-value suggests the result is unlikely due to random noise from particular participants or data points, but does not imply the result’s truth or importance.

6. Publish Informative Null Results

Prioritize publishing null results for theoretically interesting questions, especially those that contradict prior positive findings or widely held beliefs, to advance the field and avoid pursuing dead ends.

7. Fund High-Value Science

Guide science funding decisions towards research that yields fundamental theoretical results, applied results with immediate utility (e.g., curing diseases), or tools and methods that accelerate future scientific progress.

8. Rigorously Double-Check Work

When producing significant work, especially that which will be widely published or cited, rigorously double-check all calculations and data analysis to minimize errors, acknowledging the increased responsibility that comes with greater impact.

9. Adopt Unit Tests in Research

Scientists should implement unit tests (writing code specifically to test their analytical code) to catch bugs and improve the reliability of their computational work, a best practice from software engineering.

10. Detail Experimental Methods

Fully disclose experimental methods in scientific papers with as much detail as possible, recognizing that subtle, seemingly minute differences in procedures can dramatically affect results and are crucial for reproducibility.

11. Avoid Grandiose Generalizations

Scientists should approach findings with humility, avoiding grandiose pronouncements about general human behavior based on limited populations or contexts, and acknowledge the potential for results to be highly specific.

12. Assess Implementation Quality

When evaluating interventions, recognize that a poor or low-quality implementation of an otherwise effective program can lead to a failure to find an effect, making it crucial to account for implementation quality.