How can we make science more trustworthy? (with Stuart Ritchie)
Stuart Ritchie, a Lecturer at King's College London, discusses making science trustworthy, examining controversies like COVID treatments (ivermectin, vitamin D) and the reliability of IQ research. He emphasizes critical thinking and understanding research limitations.
Deep Dive Analysis
15 Topic Outline
Trust in Science and COVID-19 Pandemic Lessons
Failures in Top Medical Journals: Surgisphere Scandal
Ivermectin and Vitamin D: Case Studies in Scientific Controversy
Challenges of Controlling Variables in Observational Studies
Power of Randomized Controlled Trials and Natural Experiments
The Decline Effect in Scientific Research Findings
Revisiting Ivermectin: Advocacy, Fraud, and Nuance
Hype Cycle in Science: Power Posing and Growth Mindset Examples
Demystifying Intelligence (IQ) Research Controversies
Misconceptions About IQ Tests and Their Predictive Power
IQ, Skill Acquisition, and Compensation by Other Traits
Philosophical Interpretations of General Intelligence (G-factor)
Limitations of Current Brain-Level Understanding of Intelligence
Beyond General Intelligence: Specific Domains and Trainability
Improving Information Ecosystems: Flagging Controversial Claims
8 Key Concepts
Replication Crisis
A phenomenon in science, particularly psychology, where initial strong findings often fail to be reproduced in subsequent studies, leading to questions about the reliability of published research. It highlights issues like small sample sizes and publication bias.
Confounding Factors
Variables in observational studies that are correlated with both the independent variable (e.g., low vitamin D) and the dependent variable (e.g., bad COVID cases), making it difficult to determine a direct causal link. Researchers attempt to 'control' for these statistically.
Randomized Controlled Trial (RCT)
The 'gold standard' of scientific research, where participants are randomly assigned to either a treatment group or a control group. This randomization ensures that, on average, there are no systematic differences between groups, allowing for strong causal inferences.
Natural Experiment
A research design that leverages naturally occurring events or policy changes that randomly or quasi-randomly assign individuals to different conditions, allowing researchers to study causal effects without direct experimental manipulation.
Decline Effect
The observation that the effect sizes reported in initial studies on a topic tend to be larger and more dramatic than those found in later, often more rigorous, replication studies. This can be due to publication bias, small sample sizes, and methodological improvements over time.
Indifference of the Indicator
A concept proposed by Charles Spearman, suggesting that a good measure of general intelligence (G-factor) can be extracted from a wide variety of cognitive tests, as long as a sufficient number and variety of tests are used. The specific tests in the battery don't dramatically alter the resulting G-factor.
Bonds Theory of Intelligence
Proposed by Godfrey Thompson, this theory suggests that what appears as a general factor of intelligence (G) in test data might actually arise from multiple, uncorrelated specific mental abilities. Tasks might tap into various combinations of these abilities, creating an illusion of correlation between tests.
Near vs. Far Transfer
In the context of cognitive training, 'near transfer' refers to improvements in tasks closely related to the trained task (e.g., working memory training improving other working memory tasks). 'Far transfer' refers to improvements in broader, untrained cognitive abilities (e.g., working memory training improving general intelligence), which is rarely observed.
7 Questions Answered
It's not as simple as trusting all science or relying solely on journal reputation; listeners must consider the transparency of study plans, the quality of methods, sample sizes, and the provenance of data, as exemplified by the COVID-19 pandemic.
Observational studies struggle with confounding factors and measurement error in controls, making it difficult to isolate true causal effects, whereas randomized controlled trials inherently balance unmeasured confounders.
For both ivermectin and vitamin D, larger, higher-quality randomized controlled trials generally found no significant effect against COVID-19, despite initial enthusiasm and some early studies suggesting benefits.
IQ research is controversial due to historical abuses, political biases, and cultural ideas about equality, but the claim that IQ tests only measure test-taking ability is inaccurate; IQ scores correlate with and predict many important life outcomes to a useful extent.
Higher intelligence generally means people learn skills faster and can potentially reach higher levels of peak performance in cognitively demanding tasks like chess, but other factors like conscientiousness can compensate for lower cognitive abilities.
The general factor is the common underlying ability shared across a wide variety of cognitive tests; it is measured by administering a diverse battery of tests and extracting the shared variance, which remains consistent regardless of the specific tests used.
Specific cognitive abilities, like working memory, can be improved through targeted training (near transfer), but there is generally a lack of evidence that such training leads to improvements in broader general intelligence (far transfer).
14 Actionable Insights
1. Be Skeptical of All Claims
Remember that “people make stuff up, even people you like.” Apply critical thinking and skepticism to all information, as deliberate falsehoods or self-deception can occur.
2. Evaluate Scientific Studies Critically
Don’t blindly trust scientific claims, even from top journals. Look for transparency, justified methods, and adequate sample sizes, and be wary of studies lacking these qualities.
3. Prioritize Randomized Controlled Trials
When evaluating treatments or interventions, prioritize evidence from randomized controlled trials (RCTs) as they are the gold standard for establishing causality and are less prone to confounding factors than observational studies.
4. Expect Decline in Effect Sizes
Be aware of the “decline effect,” where initial studies often show larger effects that diminish in subsequent, more rigorous research. This means early findings may overstate the true impact.
5. Avoid Overstating Claims
When advocating for a finding or treatment, avoid making outrageously strong claims, even if you believe in it. Overstating benefits can erode trust and make your position seem less credible.
6. Trace Information to Source
When encountering claims, especially controversial ones, trace them back to their original source to verify their provenance and avoid spreading made-up stories.
7. Question “Controlled For” Claims
When researchers claim to have “controlled for” variables in observational studies, ask for specifics on how they measured and controlled those factors, as measurement error and over-controlling can invalidate conclusions.
8. Map Causal Assumptions
Before drawing conclusions from data, map out the assumed causal relationships between variables using tools like directed acyclic graphs. This practice helps reveal hidden assumptions and encourages circumspection about the data’s true causal structure.
9. Identify Controversial Claims
Seek to identify when a claim is controversial, meaning many people disagree with it, rather than just accepting it as objective fact. This awareness should prompt greater skepticism and a second thought before immediately adopting the claim as true.
10. Utilize Learning Strategies
Employ effective learning strategies from cognitive psychology to learn more efficiently, regardless of your innate intelligence level. Teachers should also equip students with these tools.
11. Cultivate Conscientiousness
Develop high conscientiousness, as it can compensate for lower cognitive abilities by enabling highly organized and productive work habits, leading to success in demanding fields.
12. Focus on Specific Skill Development
Instead of solely focusing on general intelligence, identify specific domains or skills you want to improve and actively practice them. You can get better at any particular domain through effort and dedicated practice.
13. Understand IQ as Learning Potential
View IQ as a measure of potential to pick up and learn new skills faster, rather than an absolute limit. Higher IQ can accelerate skill acquisition, but lower IQ does not prevent learning, it just may take longer.
14. Broaden Social Exposure
Actively seek experiences and interactions with people across the full range of intelligence and educational backgrounds. This can help align your subjective understanding of intelligence with the broader evidence.
3 Key Quotes
It does create this real issue, which is, if people say trust the science, well, which science, right? There's good science and there's bad science. And it's not like so clearly delineated, right? It's not like all bad science is published in bad science journals and all good science is published in good science journals. Like it really actually takes some nuance to tell what is what.
Stuart Ritchie
People make stuff up, even people you like. That's my, that's like my, like, people always are constantly making stuff up, even people you like. Like, that's like the bottom line for critical thinking for me.
Stuart Ritchie
The predictive ability of IQ tests exists regardless of what the interpretation of the general factor is, right? There's the predictive validity when you have these, you just have these tests, and then you just see what predictions you can make. And then trying to understand why that is the case is a different question.
Stuart Ritchie