How broken is social science? (with Matt Grossman)
Spencer Greenberg and Matt Grossman discuss the evolving quality and reproducibility of social science. Grossman argues it's never been more rigorous, relevant, or self-reflective, despite inherent challenges, highlighting progress in methods, data, and diversity.
Deep Dive Analysis
14 Topic Outline
Introduction to Social Science Challenges and Progress
Increased Rigor in Social Science Research
Predictive Power in Political Science
Causal Inference and Descriptive Data in Social Science
The Replication Crisis and Learning from Past Studies
Increasing Relevance of Social Science
Self-Reflection and Methodological Debates like Pre-registration
Accumulation of Knowledge in Social Science
Social Science Contributions During the COVID Pandemic
Diversity and Political Bias in Social Science
Critiques and Evolution of the University System
Challenges and Opportunities for PhD Students
Optimism Versus Pessimism in Social Science Progress
Incentives and Structural Problems in Academia
5 Key Concepts
Causal Inference Revolution
A movement, originating in economics and spreading through social sciences, that focuses on using experimental designs and advanced statistical techniques (like instrumental variables or regression discontinuity) to establish more definitive cause-and-effect relationships in specific situations.
Pre-registration
A practice where researchers submit a plan outlining their analyses and data collection methods before conducting a study. This aims to increase transparency and reduce practices like p-hacking or post-hoc theorizing.
Statistical Power
The probability that a study will find a statistically significant effect if an effect truly exists. Low statistical power means there is a high chance of missing real effects, even if they are present.
Replication Crisis
The phenomenon where many previously published scientific findings, particularly in social psychology, fail to produce the same results when re-tested. This raises questions about the reliability and generalizability of past research.
Racial Resentment Scale
A political science scale designed to measure racial attitudes, but whose name and interpretation often imply a negative valence. This can potentially oversimplify or mischaracterize the range of perspectives held by individuals on the scale.
8 Questions Answered
Yes, social science is becoming more rigorous with better data, methods, and causal inference strategies; more relevant by engaging with real-world problems; and more self-reflective by openly discussing methodological issues and biases.
While it's still difficult to predict complex, far-off events, social science models are improving over time, especially for more predictable outcomes like election results in most districts.
Historically, researchers may not have been fully aware of metascience findings regarding statistical power, leading to persistent use of small samples. This issue is now being addressed through collective learning and increased awareness.
A single paper today is likely more rigorous than its predecessors, but science is a social enterprise. Skepticism is warranted, especially regarding generalizability, as results can be brittle and sensitive to subtle changes in population or context.
Social science does accumulate knowledge, albeit slowly. The COVID pandemic demonstrated its ability to understand social factors influencing health outcomes and policy implementation, even if immediate actionable success is hard to achieve.
Social science is overwhelmingly composed of people on the left, which can lead to different questions being asked, more negative views of political opponents, and less generous interpretations of evidence related to conservative values and beliefs.
While there's a consensus among scholars and the public about perceived university problems (e.g., bureaucracy, job market for PhDs), these issues are not seen as inhibiting the development or accumulation of social science research, which continues to grow and expand its reach.
While it's difficult for individuals to interpret and apply general research evidence to their specific lives, it is possible for people to improve their lives, as demonstrated by successful applications like cognitive behavioral therapy.
23 Actionable Insights
1. Increase Diversity in Social Science
Actively incorporate more women, racial minorities, and international perspectives into social science research to broaden topics, diversify interpretations, and uncover biases from past research.
2. Include Diverse Political Perspectives
Ensure conservatives remain part of science and engage in the process, commenting on research to mitigate biases that arise from an overwhelmingly liberal social science community and lead to less generous interpretations of political opponents.
3. Self-Reflect on Political Biases
Researchers should critically examine how their own political perspectives, such as being liberal, might be influencing their investigations and interpretations of evidence, especially on politically charged topics.
4. Adopt Different Thinking Habits for Predictions
To improve predictions, individuals should be taught to adopt different thinking habits, as broad knowledge of general social trends can be more effective than deep saturation in one area of expertise.
5. Work in Groups for Better Predictions
Collaborate in groups to make better predictions, as team science and worldwide interactions in interpreting social events can help protect against basic human biases.
6. Narrow Scope of Claims
Be humble and narrow the scope conditions of your claims, avoiding broad pronouncements or reducing complex historical processes to a few variables, which is a productive trend in social science.
7. Leverage Others to Identify Biases
Recognize that while we are not great at understanding our own biases, we are very good at identifying others’ biases, so engage with evidence collected by other people to more closely approximate the truth through a social process.
8. Embrace Unlearning Falsehoods
View the discovery of problems in past studies and the unlearning of previously held beliefs as progress, as this process of re-evaluation regularly helps refine the truth of claims.
9. Be Cautious with Popularized Research
Exercise caution when consuming popularized research, as the journey from scholarship to popularization often involves skipping verification steps, making it more likely to contain unverified claims.
10. Engage in Public Debate on Claims
Foster public debate about big claims, as this process allows for pushback with different kinds of evidence, helping to refine the truth and narrow the application of broad claims.
11. Distinguish Exploratory from Confirmatory Research
Be clear about whether research is exploratory or designed for confirmation of theory, as recognizing this distinction is crucial for evaluating the findings and their implications.
12. Utilize Pre-registration for Studies
Consider adopting pre-registration for studies, where a plan for analyses and data collection is submitted beforehand, as it is a rising intervention that improves rigor despite its challenges.
13. Foster Cross-Field Interaction
Promote more translation and interaction between different fields, such as media, think tanks, academia, and politics, to flatten attempts to understand the world and make it easier for scholars to find and recognize evidence.
14. Inform Public Conversations with Evidence
Make an effort to respond to real-world public conversations and bring the best available evidence to inform those discussions, increasing the relevance of social science.
15. Diversify Evidence & Decision-Makers
Incorporate diverse types of evidence and include different types of people in the decision-making chain when applying research findings, especially for actionable interventions.
16. Acknowledge Historical/Contextual Bias
Be aware that research findings from one time period or culture may not apply to others, as the diversity of human experience means causes and outcomes can differ across time and populations.
17. Utilize Evidence-Based Self-Improvement
Seek self-improvement strategies that are rooted in repeated research results and have been successfully translated into applied forms, such as cognitive behavioral therapy.
18. Engage with Real-World Practitioners
Recognize that innovations often come from the engagement of academic professions with actual actors in the field, rather than solely from within the profession itself.
19. Value Critical Perspectives
Acknowledge and value the perspectives of critics, as their critiques, even if disagreed with, are crucial for stimulating broad efforts and driving the development and improvement of social science.
20. Change Institutions & Share Information to Improve Behavior
Recognize that even when actors have incentives to avoid progress, institutional changes and making people aware of new information can lead to improved behavior and outcomes.
21. Publish Data Sets
Publish data sets alongside research to promote transparency and allow others to replicate findings and build on the science.
22. Implement Pre-publication Data Checking
Utilize pre-publication data checking in journals to ensure the rigor and reproducibility of research before it is published.
23. Learn from Research Practices
Engage in a continuous learning process from research practices, such as pre-registration or data checking, to better understand the nature of research and refine methodologies.
6 Key Quotes
It's definitely inherently more difficult to understand the human experience in a scientific way than it is for some other parts of science.
Matt Grossmann
Humbleness is part of the learning here. Yeah, it is. It's to narrow the scope conditions of your claims, to try to not make a sort of world historical process into a few variables.
Matt Grossmann
Today, when I look at a paper in a top journal, I assume there's about a 40% chance it won't replicate.
Spencer Greenberg
Unlearning some things that we thought were true before, but don't turn out to be true or don't turn out to be as true as to the extent that they were initially published is still a learning process and one that we are going through regularly.
Matt Grossmann
I think that one paper would be better than its equivalents 10 or 15 years ago by a pretty considerable margin.
Matt Grossmann
It's very hard to come up with something important and novel and true, right? That's a really tall order. But if you need seven publications in top journals to get a job in the field, and almost nobody except like the greatest, you know, super geniuses can come up with seven important novel, true ideas, then something's got to give like, there's going to be a pressure towards doing shoddy research.
Spencer Greenberg