How to measure impact, and why we may have all been doing it wrong (with Michael Plant)
Spencer Greenberg speaks with Michael Plant, founder of the Happier Lives Institute, about using subjective well-being (happiness and life satisfaction) as the primary metric for evaluating interventions. They discuss the surprising finding that psychotherapy can be significantly more cost-effective than cash transfers in improving well-being, and critique traditional prioritization frameworks.
Deep Dive Analysis
13 Topic Outline
Introduction to Happiness as a Measure of Good
Measuring Subjective Well-being: Reliability and Validity
Challenges and Nuances in Happiness Measurement
Critique of QALYs and DALYs as Wellbeing Metrics
Disconnect Between Expected and Actual Happiness Impacts
Evolution of Cost-Effectiveness in Effective Altruism
Critique of Scale, Neglectedness, and Tractability Framework
Comparing Psychotherapy to Cash Transfers in Low-Income Countries
Future Research Directions for Happier Lives Institute
Bhutan's Gross National Happiness Metric
Cross-Cultural Comparability of Happiness Measures
Improving and Leveraging Happiness Measures
The Role of Philosophy in Doing Good Better
7 Key Concepts
Subjective Well-being (SWB)
SWB is a social science term referring to self-reported measures of happiness, life satisfaction, and sometimes meaning in life. A common measure is asking individuals to rate their overall life satisfaction on a scale, like zero to ten.
Construct Validation
This philosophical concept tests if a measure (like happiness) truly captures what it intends to. It involves observing how the measure correlates with expected real-world outcomes, such as higher income, better health, or lower suicide rates, to build confidence in its validity.
Easterlin Paradox
This finding in happiness research suggests that while richer individuals and countries tend to be happier than poorer ones at any given time, rich countries, on average, do not get happier over time despite economic development. This creates a tension with the expectation that societal progress should lead to increased happiness.
Quality-Adjusted Life Years (QALYs)
QALYs are standardized health metrics that quantify the value of health interventions. They are calculated by asking members of the public to make time trade-offs between years lived with a specific health condition and years of healthy life, assigning a 'quality' score to different health states.
Disability-Adjusted Life Years (DALYs)
DALYs are similar to QALYs but measure years of 'unhealthy' life lost due to disease or disability. While QALYs are a positive measure of healthy life, DALYs are a reverse quality, quantifying the burden of disease.
Affective Forecasting Failures
These are psychological biases where people inaccurately predict how future events or conditions will affect their emotional state or happiness. This can lead to a disconnect between what people expect to affect happiness and what actually does, especially regarding health conditions.
Task Shifting in Therapy
This approach involves training non-specialists, such as lay health workers, to deliver psychotherapy instead of highly trained mental health specialists. Research suggests that this method can be nearly as effective as specialist-led therapy, particularly in low-income country contexts, making interventions more scalable and cost-effective.
8 Questions Answered
Michael Plant advocates for an approach centered on maximizing happiness, which he defines as experiencing overall positive feelings. He believes that subjective well-being measures, like self-reported life satisfaction, are crucial for identifying the most effective interventions.
Self-reported measures of happiness and life satisfaction are considered reasonably reliable because they correlate with expected real-world indicators like smiling, income, health, and are negatively associated with things like suicide. While not perfect, they behave in a believable way, giving confidence in their ability to measure happiness.
While some things are more important for one measure than the other (e.g., mental health for happiness, income for life satisfaction), recent research comparing psychotherapy and cash transfers found that the specific measure used (happiness, life satisfaction, or standardized mental health) did not make a significant difference in the estimated impact.
The main flaws are that they only capture health, not overall well-being, making it hard to compare health improvements to other interventions like income or education. Additionally, they rely on people guessing how bad a condition is, which can be inaccurate due to failures of affective forecasting, rather than actual lived experience.
People often fail at 'affective forecasting,' meaning they inaccurately predict how certain conditions will affect their happiness. Conditions that people adapt to, like walking with a limp, might be less impactful on long-term happiness than expected, while persistent issues like mental anxiety or physical pain, which are harder to adapt to, can have a greater, sustained negative impact.
This framework is problematic because it's confusing and often leads to an implicit comparison of cost-effectiveness anyway. It's difficult to assess these factors in the abstract without considering specific actions, and ultimately, the goal is to find actions with the highest expected value, which combines these elements directly.
Research by the Happier Lives Institute suggests that psychotherapy, particularly interpersonal group therapy delivered by non-specialists in low-income countries (like Strong Minds), is about 12 times more cost-effective than monthly cash transfers (like GiveDirectly) in improving subjective well-being. This is due to therapy's larger initial effect and significantly lower cost, despite cash transfers potentially lasting longer.
Yes, research suggests that happiness scales are largely comparable across cultures. A study of immigrants to Canada from over a hundred countries found that their average life satisfaction levels and distributions converged with those of Canadians, indicating a consistent understanding and use of the scales despite diverse cultural backgrounds.
13 Actionable Insights
1. Support Mental Health Interventions
Prioritize funding and implementing mental health interventions, specifically group psychotherapy delivered by non-specialists in low-income countries, as research indicates they are significantly more cost-effective (e.g., 12x) than cash transfers for improving well-being.
2. Prioritize Subjective Well-being
Shift from proxy measures like health (QUALYs/DALYs) or income to direct self-reported happiness and life satisfaction when evaluating the impact of interventions, as these are closer to what ultimately matters.
3. Retire SNT Framework
Abandon the “Scale, Neglectedness, Tractability” framework for cause prioritization and instead directly evaluate the expected value and cost-effectiveness of specific, actionable interventions.
4. Model Uncertainty with Simulations
When evaluating intervention effectiveness, use Monte Carlo simulations to account for uncertainty in various factors, leading to more robust and confident conclusions about impact.
5. Focus on Chronic Mental Suffering
Prioritize interventions that address chronic mental anxiety and physical pain, as these are conditions people do not adapt to and consistently reduce happiness, potentially being overlooked by other measures.
6. Advocate for Happiness in Policy
Advocate for governments to adopt happiness as the ultimate measure for judging public policy, polling people on a 1-10 scale and using this data to guide decisions for long-term societal well-being.
7. Evaluate Personal Project Value
When considering a personal project, ensure it offers unique value (neglectedness), aligns with your skills (tractability), and has the potential for significant positive impact if successful (scale).
8. Collect Detailed Happiness Data
Prioritize collecting more detailed happiness data using methods like the experience sampling method to better understand people’s moment-by-moment enjoyment of life, rather than solely relying on life satisfaction.
9. Trust Cross-Cultural Happiness Data
Rely on cross-cultural comparisons of self-reported happiness scales, as research suggests a consistent understanding and use of these scales across different cultures.
10. Leverage Philosophical Rigor
Apply philosophical rigor to clearly define what truly matters and use clear-headed analysis to determine the most effective and practical actions to achieve those goals.
11. Investigate Micro-Interventions
Research and potentially support micro-interventions such as the Friendship Bench mental health charity, mental health apps, cataract surgery, fistula surgery, and providing cement flooring, as these show promise for high cost-effectiveness.
12. Explore Meso-Interventions
Investigate meso-level interventions including researching psychedelics for mental health treatment, improving access to opiates for pain relief in poor regions, and evaluating the impact of changes in recreational drug laws.
13. Support Happier Lives Institute
Consider donating to the Happier Lives Institute to fund ongoing research into effective ways to improve global well-being, particularly in areas like mental health interventions.
4 Key Quotes
Ultimately, there isn't a sort of a single mechanical test you can do to assess whether your measure of something is valid, which is to say that actually captures what it's trying to capture. Ultimately, it's kind of an evaluation.
Michael Plant
The problem is you have this subjective thing we're trying to capture. So someone's happiness or their life satisfaction or their sense of meaning in life. And like, obviously, this isn't an objective thing. So you can't go out and like prod it.
Michael Plant
The only sensible way to compare causes is to compare like particular things you can do to make progress on them.
Michael Plant
I think the role of philosophers is to think really hard about what matters and then work out what we should do next as a result.
Michael Plant