#143 - John Ioannidis, M.D., D.Sc.: Why most biomedical research is flawed, and how to improve it
This episode features physician-scientist and Stanford professor John Ioannidis, a meta-research expert. He discusses why most published research findings are false, particularly in nutritional epidemiology, and offers insights into improving scientific rigor and communication.
Deep Dive Analysis
12 Topic Outline
John Ioannidis's Background and Approach to Science
Mathematical Model: Why Most Published Research Findings Are False
Understanding P-values and Statistical vs. Clinical Significance
The Problem of Power in Research: Underpowered and Overpowered Studies
Contrasting Research Practices: Genetics vs. Nutritional Epidemiology
Strategies to Improve Nutritional Epidemiology Research
Impact of Pre-existing Beliefs and Non-Financial Conflicts in Science
Questionable Research Practices and the Need for Better Standards
Funding Science: Roles of Public, Private, and Philanthropic Sectors
Challenges in Epidemiology: The PREDIMED Study Case
SARS-CoV-2 Seroprevalence Study and Politicization of Science
Optimism for the Future of Science and Knowledge
8 Key Concepts
Meta-research
Meta-research is the study of scientific research itself, examining its methods, reporting, reproducibility, and validity. It aims to understand and improve the credibility and efficiency of the scientific enterprise.
P-value
A p-value is a statistical measure that indicates the probability of obtaining a result as extreme or more extreme than the one observed, assuming the null hypothesis is true. Traditionally, a p-value less than 0.05 is considered 'statistically significant,' but this threshold is often misinterpreted and does not account for bias or clinical relevance.
Statistical vs. Clinical Significance
Statistical significance refers to the likelihood that a result is not due to random chance, often determined by a p-value. Clinical significance, however, refers to whether a finding is large enough and meaningful enough to make a practical difference for patients or public health, considering costs, harms, and alternatives.
Power (in studies)
Statistical power refers to a study's ability to detect a true effect if one exists. Underpowered studies (too small) are likely to miss true effects and, when they do find a significant result, the estimated effect size is often exaggerated and more likely to be a false positive. Overpowered studies (too large, often with 'big data') can find statistically significant results that have no clinical meaning and may simply be measuring bias.
Bias
Bias in research refers to systematic errors that can lead to results appearing statistically significant when they should not be. It can take many forms, including publication bias (only positive results are published), selective reporting bias (only favorable outcomes are highlighted), confounding bias, and information bias, all of which distort the true findings.
Bradford Hill Criteria
These are a set of considerations proposed by Sir Austin Bradford Hill to help determine if observed associations in epidemiology are causal. They include strength, consistency, specificity, temporality, biological gradient, plausibility, coherence, experiment, and analogy, though Hill himself emphasized they are not rigid rules and no single criterion is bulletproof.
Mendelian Randomization
Mendelian randomization is a research method that uses genetic variants as instrumental variables to infer causal relationships between a modifiable risk factor and a disease outcome. By leveraging the random assortment of genes during meiosis, it can create designs that are somewhat equivalent to randomized trials, helping to reduce confounding inherent in observational studies.
Exposure-wide Association Testing
This is an approach to analyzing observational data that involves simultaneously testing and reporting on all collected exposures (e.g., nutrients) and outcomes, rather than focusing on one at a time. It aims to provide a more transparent and complete picture of associations, accounting for multiplicity and correlations, to identify recurrent and replicable signals.
7 Questions Answered
A mathematical model suggests that most published research findings are false due to a combination of factors including low prior probability of a true effect, underpowered studies, various forms of bias (e.g., publication, selective reporting), and the large number of scientists independently testing hypotheses, which increases the chance of false positives.
In an underpowered environment, even if true signals are detected, their estimated effects are often exaggerated compared to their true magnitude. Across a field with many underpowered studies, any detected statistically significant signal is more likely to be a false positive rather than a true effect.
Genetics benefited from better, more accurate measurement tools (decoding the human genome), fewer strong pre-existing beliefs or 'experts' fighting for specific gene variants, and the adoption of more stringent statistical significance thresholds (e.g., p-value of 10^-8). The field also embraced large-scale data sharing, common protocols, and standardized analyses across multiple research teams.
Nutritional epidemiology often suffers from highly inaccurate measurement tools (like questionnaires with high recall bias), strong prior beliefs and expert opinions that can influence interpretation, and a tendency for post-hoc research that is not pre-registered and is selectively presented, leading to 'data torturing' to find significant associations.
Improvements can come from conducting more controlled experimental trials (even under supervised conditions for efficacy), utilizing Mendelian randomization studies to infer causality, and adopting 'exposure-wide association testing' to analyze all nutrients and outcomes transparently while accounting for multiplicity and correlations.
Philanthropy can play a crucial, catalytic role by providing no-strings-attached funding for high-risk, innovative ideas that might be difficult to fund through traditional public or private channels. This allows scientists to pursue potentially groundbreaking research without the pressure of immediate financial returns or political deliverables.
These studies, conducted in early months of the pandemic, indicated that the virus was far more widely spread than suggested by the number of documented PCR cases, with seropositivity being about 50 times more common. This implied a significantly lower infection fatality rate per infected person than initially believed, alongside a steep risk gradient where some populations (e.g., nursing home residents) faced tremendous risk.
22 Actionable Insights
1. Embrace Not Knowing
Actively seek out what you don’t know and be open to correcting previous understandings, as this continuous learning and self-correction are fundamental for scientific and human progress.
2. Prioritize Clinical Significance
When evaluating medical interventions or research findings, focus on whether an action makes a meaningful difference to a patient or community, rather than solely on statistical significance.
3. Adopt Genetics Research Model
For fields like epidemiology, conduct very large studies, analyze all relevant factors, join forces with other researchers to share data and protocols, and standardize analysis to maximize power, transparency, and credibility.
4. Pre-specify & Register Hypotheses
For randomized trials and other research, carefully pre-specify and register your hypotheses and protocols, ensuring they address clinically important and meaningful effect sizes, to establish transparent rules for the study.
5. Be Transparent About Biases
Recognize that scientists are human and have beliefs; strive to map these beliefs, be transparent about them, and actively restrain their influence on research conduct and interpretation to maintain objectivity.
6. Prioritize Experimental Evidence
When seeking to establish causality, prioritize obtaining experimental evidence, such as well-designed randomized trials, as this is the most reliable approach.
7. Use Stringent P-Value Thresholds
In scientific fields, particularly those with high multiplicity, consider using a more stringent p-value threshold (e.g., 0.005 or 10^-8) instead of the traditional 0.05 to reduce false positives.
8. Avoid Underpowered Studies
Do not conduct small, underpowered studies, as they are likely to produce false positive results or grossly exaggerate the magnitude of any detected signals.
9. Be Cautious with Big Data
When working with large datasets, be aware that studies can be overpowered, leading to statistically significant results that lack clinical meaning and may simply be measuring bias.
10. Implement Safeguards Against Self-Deception
Actively impose safeguards throughout the research process to minimize the chances of fooling yourself, which is the most important rule in science for maintaining honesty and integrity.
11. Conduct Exposure-Wide Association Testing
Instead of reporting on one nutrient or exposure at a time, analyze all collected exposures and outcomes simultaneously, reporting results that account for multiplicity and correlation, to achieve a more transparent and complete picture.
12. Utilize Mendelian Randomization
Consider using Mendelian randomization studies, which leverage genetic instruments, to create designs that are functionally equivalent to randomized trials, thereby enhancing the credibility of signals from observational research.
13. Verify with Multiple Analyst Teams
To enhance the credibility of research findings, especially in fields like nutrition, employ two or three independent analyst teams to analyze the same data and confirm consistent results.
14. Interpret Bradford Hill Cautiously
When applying Austin Bradford Hill’s criteria for causality, do so with temperance and caution, understanding that none of them are bulletproof rules and that nature’s operations can be unpredictable.
15. Consider Real-World Adherence
When evaluating dietary or other interventions, recognize that a diet or treatment is not truly better if people cannot adhere to it in real-life circumstances, as adherence is part of effectiveness.
16. Conduct Controlled Efficacy Trials
To gain insights into efficacy, run randomized trials under very controlled, supervised circumstances (e.g., in a clinic) with stringent monitoring of diet and physiological responses, to understand optimal treatment under ideal conditions.
17. Advocate for Better Research Incentives
Work to realign incentives in science away from producing ‘striking’ results from small studies and towards supporting high-risk, definitive research, even if it has a high chance of failing, to truly advance knowledge.
18. Communicate Science Honestly
Communicate scientific findings to the public with honesty, clarity, and without exaggerated promises, defending scientific methods and principles against non-scientific claims and populist attacks.
19. Protect Scientists from Attacks
Advocate for protecting scientists from personal attacks and vitriol, ensuring they can disseminate their objective scientific findings without fear of reprisal, especially in highly polarized environments.
20. Learn Mathematics
Cultivate a love for and understanding of mathematics, as it forms the foundation for many scientific endeavors and can transform approaches to complex questions, enabling significant progress.
21. Combine Rigorous Science & Quantitative Tools
In medicine and other fields, integrate rigorous scientific methods with quantitative approaches and tools to obtain reliable evidence and make a meaningful difference for human beings.
22. Continuously Reinvent Self as Scientist
Embrace the role of a scientist as a continuous process of reinvention, constantly searching for new frontiers, new questions, and new ways to correct errors and improve upon previous understanding.
6 Key Quotes
I think that intellectual curiosity is very interesting, but the ability to make a difference for human beings and to save lives, to improve their quality of life seemed to be at least in my eyes as a young person, something that was worthwhile pursuing.
John Ioannidis
If you don't fail, you're not going to succeed.
John Ioannidis
The most important rule in science is not to fool anyone. And that starts with yourself. You're the most, you're the easiest person to fool. And once you fooled yourself, the game is over.
Peter Attia
Science is the best thing that has happened to humans.
John Ioannidis
I'm an expert, but I don't know. And this is why I believe that we need to know because these are questions that could make a difference for you.
John Ioannidis
We are just at the beginning of, of knowledge. And I feel like a little kid who just wants to learn a little bit more, a little bit more each time.
John Ioannidis