#188 - AMA #30: How to Read and Understand Scientific Studies
Peter Attia and Bob Kaplan discuss how to critically evaluate scientific studies and distinguish signal from noise. They cover various study types, clinical trial phases, potential biases, and Peter's personal process for reading scientific papers.
Deep Dive Analysis
11 Topic Outline
Introduction to Studying Studies and Sifting Noise
Process for a Study: From Idea to Design and Execution
Types of Studies: Observational vs. Experimental
Observational Studies: Case Reports, Case Series, and Cohort Studies
Experimental Studies: Non-Randomized and Randomized Trials
Systemic Reviews and Meta-Analyses
Phases of Human Clinical Trials: Phase One
Phases of Human Clinical Trials: Phase Two
Phases of Human Clinical Trials: Phase Three
Phases of Human Clinical Trials: Phase Four (Post-Marketing)
Factors Affecting Confidence in Observational Studies
12 Key Concepts
Null Hypothesis
This is the starting position in scientific inquiry, stating that there is no relationship or effect between two phenomena being studied. For example, drinking coffee does not change eye color.
Alternative Hypothesis
This is the counter-position to the null hypothesis, suggesting that there is a relationship or effect between the two phenomena. For example, drinking coffee does change eye color.
Power Analysis
A crucial step in experimental design that determines the necessary number of subjects for a study. This ensures the study is adequately sized to detect a real effect if one exists.
Institutional Review Board (IRB)
A committee responsible for approving the ethics of any study involving human or animal subjects. Approval is required before a study can commence.
Observational Studies
Studies where researchers observe patterns and associations without intervening or assigning treatments. Examples include individual case reports, case series, and cohort studies, which can be retrospective or prospective.
Experimental Studies
Studies where researchers actively intervene and assign treatments to different groups to test a hypothesis. These can be non-randomized or, ideally, randomized controlled trials to minimize bias.
Meta-analysis
A statistical technique used to combine data from multiple studies that address the same research question. Each study is weighted based on its precision, with larger studies often given more weight.
Investigational New Drug (IND) Application
A formal application filed with the FDA after preclinical animal studies, signaling the intention to test a new drug in human subjects. This is the first step before human clinical trials begin.
Phase One Clinical Trial
The initial phase of human clinical trials, typically involving a small number of patients, focused on dose escalation to determine the drug's toxicity and safety profile across various doses. Efficacy is not the primary goal.
Phase Two Clinical Trial
Following Phase One, these trials continue to assess safety while also beginning to look for signs of efficacy. They are often open-label and may or may not include a control arm, typically involving tens to a few hundred patients.
Phase Three Clinical Trial
A large-scale, rigorous trial (often involving thousands of patients) that is typically randomized, blinded, and placebo-controlled (or compared to standard of care). This phase provides the gold standard data for measuring efficacy and safety for drug approval.
Phase Four Clinical Trial
Also known as post-marketing studies, these trials occur after a drug has been approved and is on the market. They gather additional safety information from a broader patient population or explore new indications for the drug.
3 Questions Answered
A study typically starts with a hypothesis (often a null hypothesis), followed by experimental design, determination of sample size through power analysis, obtaining Institutional Review Board (IRB) approval, defining primary and secondary outcomes, developing a statistical plan, pre-registering the study, and securing funding.
Studies broadly fall into observational (e.g., case reports, case series, cohort studies) and experimental categories (e.g., non-randomized and randomized controlled trials). Observational studies involve observing without intervention, while experimental studies involve active intervention and treatment assignment.
Human clinical trials progress through four phases: Phase One focuses on dose escalation and safety in a small group; Phase Two continues safety evaluation and begins to look for efficacy; Phase Three is a large, rigorous trial for definitive efficacy and safety data for approval; and Phase Four (post-marketing) gathers additional information after approval.
10 Actionable Insights
1. Scrutinize Meta-Analyses’ Constituent Studies
Do not accept a meta-analysis as definitive without examining its individual studies, because a meta-analysis of poor-quality studies will yield poor-quality conclusions, as “garbage in, garbage out” and cannot clean garbage.
2. Prioritize Rigorous Randomized Trials
When evaluating scientific literature, rely more on a single, rigorous randomized controlled trial (RCT) that is ‘worth its salt’ over a meta-analysis that may include many lower-quality studies, as the quality of individual studies is paramount.
3. Identify Randomized Controlled Trials
Look for randomized controlled trials (RCTs) as the “gold standard” for studying questions, as they attempt to remove bias by randomly assigning people into treatment groups.
4. Observational Studies Lack Causality
Understand that observational studies, whether retrospective or prospective, can only identify patterns or associations, but cannot establish causality because researchers are merely observing without intervention.
5. Beware Non-Randomized Trials
Be cautious of non-randomized controlled trials, as the lack of random assignment means there’s a reason subjects are in specific groups, which will undoubtedly introduce bias and limit the study’s conclusions.
6. Understand Case Report Limitations
Recognize that individual case reports, while valuable for generating hypotheses and highlighting rare observations, are not generalizable to broader populations and cannot comment on the efficacy of any intervention.
7. Phase 1 Drug Trial Purpose
Know that Phase 1 clinical trials are very small studies (typically less than a hundred people) focused solely on dose escalation to determine drug toxicity across a range of doses, not on whether the drug works or provides patient benefit.
8. Phase 2 Drug Trial Purpose
Understand that Phase 2 clinical trials continue to assess safety and begin to look for efficacy, but are often done in an open-label (non-randomized) fashion, which can introduce bias.
9. Phase 3 Drug Trial Rigor
Recognize Phase 3 clinical trials as rigorous, large-scale (potentially thousands of patients), randomized, and blinded studies that serve as the gold standard for measuring a drug’s efficacy and safety for broad approval.
10. Phase 4 Drug Trial Purpose
Be aware that Phase 4 (post-marketing) studies occur after a drug has been approved to gather additional safety information from a larger, more diverse population and to explore new indications for the drug.
2 Key Quotes
a thousand sows ears makes not a pearl necklace.
James Yang (quoted by Peter Attia)
garbage in, garbage out.
Peter Attia