The most important century (with Holden Karnofsky)
Holden Karnofsky, a Member of Technical Staff at Anthropic and co-founder of GiveWell and Open Philanthropy, discusses whether society is at "peak progress," the dynamics of innovation, and the radical uncertainty of an AI-driven future. He explores the implications of exponential growth, the "low-hanging fruit" theory, and the critical need for careful AI development and pandemic preparedness.
Deep Dive Analysis
10 Topic Outline
The Special Nature of Our Current Era
Limits to Exponential Growth and Innovation Stagnation
Evaluating Historical and Present Intellectual Productivity
Technology's Mixed Impact on Quality of Life
Risks from Advanced Technologies and Unlearned Lessons
Navigating AI Risks: Policy, Technical, and Personal Shifts
AI Alignment, Emergent Goals, and Rapid Advancement
Holden's Approach to Learning and Decision-Making
Effective Altruism: Values, Impact, and Neglected Areas
Update on Psychology's Reproducibility Crisis
5 Key Concepts
Special Time in History
The current era is unique due to unprecedented global economic growth and technological acceleration over the last 200-300 years, making it difficult to predict the future based on past trends. This period contrasts sharply with previous millennia of very low growth.
Low-Hanging Fruit Dynamics
This principle suggests that innovation becomes harder and less productive per researcher over time in any given field because the easiest and most foundational discoveries are made early on. This explains why early innovators often have outsized impact and why per-head output decreases as a field matures.
Success Without Dignity
This concept describes a scenario where humanity achieves a good outcome with advanced AI despite a 'pathetic response' or lack of responsible, coordinated effort from society. This could involve AIs being 'human-like' in their values and capabilities, or rapid technical progress in safety measures during a short window after AGI development.
Learning by Writing
This is a personal learning methodology where one starts by articulating their current opinion on a topic, then uses that as a basis to identify research questions, read critically, and seek external critiques to refine and update their understanding. This is contrasted with extensive reading before forming an opinion.
Evolution as Misaligned AI
This analogy suggests that natural selection, like a programmer, 'trained' organisms (humans) to maximize offspring. However, humans developed emergent goals (happiness, avoiding children) that are not perfectly aligned with the 'programmer's' original objective, illustrating how AI could develop unintended goals.
11 Questions Answered
The last 200 years since the Industrial Revolution have seen unprecedented global economic growth and technological change, clumping significant events and accelerating growth rates far beyond any previous period in human history.
No, a few percent annual growth for 10,000 years would require cramming hundreds of times the world's current economic value into every atom in our galaxy, which is physically impossible due to constraints like the speed of light.
This is explained by 'low-hanging fruit' dynamics; early in a field, the easiest and most foundational discoveries are made, making subsequent breakthroughs harder and requiring more resources per unit of output.
Evidence suggests quality of life may have worsened, with increased violence (homicides per capita) and potentially poorer nutrition (indicated by height) in sedentary agricultural societies compared to roving hunter-gatherer bands.
He uses a 'learning by writing' method, starting by articulating his current opinion, then identifying research questions, reading critically to challenge his views, and seeking external critiques to refine his understanding.
Anthropic focuses on designing a responsible scaling policy, securing systems for more powerful AI, and planning to reduce the risk of extreme human abuse of AI, in addition to the risk of AI taking over for itself.
This threshold is when AI can perform the kind of work done by the best human AI research teams. It's significant because once AI can do AI research on its own, it could lead to extremely rapid, self-accelerating improvements in AI capabilities.
Humans can be seen as 'misaligned AI' from natural selection's perspective; while evolution 'programmed' us to maximize offspring, we developed emergent goals (like happiness or avoiding children) that diverge from this original objective, illustrating how AI could develop unintended goals.
He is skeptical of philosophy's methodology and its ability to provide definitive answers on what we should value. He gives little weight to philosophical thought experiments when they conflict with his intuitions, heuristics, or 'juju.'
Yes, Holden believes areas like AI safety, pandemic preparedness/biosecurity, animal suffering on factory farms, and global poverty are still massively neglected and offer incredible opportunities for impact.
While about 40% of top psychology papers 15 years ago might not replicate, current replication efforts show a much lower failure rate (e.g., 2 out of 12 for Clearer Thinking's project). However, new issues like 'importance hacking' and incorrect statistics are still observed.
20 Actionable Insights
1. Embrace Epistemic Humility
Practice admitting when you are wrong to build resilience and improve long-term well-being and epistemic accuracy. Cultivate a preference for discovering and correcting errors, even if initially embarrassing, over maintaining false beliefs.
2. Learn by Writing
Adopt a “learning by writing” approach by articulating your current opinion first, then using it to identify research questions and guide your learning. Externalize and rigorously test your ideas by writing them down, outlining evidence, and actively seeking counterarguments before fully committing to a stance.
3. Read Non-Fiction Strategically
Read non-fiction strategically by skimming, focusing on introductions, and engaging with criticisms to guide deeper dives into relevant sections. This approach helps retain more information relevant to your specific interests.
4. Form Flexible Opinions
Form opinions early, even with limited knowledge, but hold them with varying strengths and be open to easy revision. Distinguish between forming an opinion (acceptable with limited knowledge) and acting forcefully or making high-stakes decisions based on it (requires more expertise).
5. Vet Expert Knowledge
Develop a strategy for identifying trustworthy experts by doing foundational research yourself, then deferring to them on specific topics. Adjust your learning investment based on the field’s complexity and your need for informed judgment.
6. Acknowledge Future Uncertainty
Acknowledge the radical uncertainty of the future by understanding historical growth patterns, and avoid extrapolating future trends solely from recent history. Recognize the current era as historically unique due to rapid technological and quality-of-life changes.
7. Plan for Stagnation
Prepare for potential stagnation or collapse rather than assuming continuous exponential growth, as indefinite multi-percent growth is unsustainable. Understand that innovation per researcher tends to decrease over time due to “low-hanging fruit” being picked.
8. Accelerate Innovation via Minds
To accelerate innovation, significantly increase the number of minds working on problems, potentially through AI, to outweigh low-hanging fruit dynamics.
9. Nuance Progress’s Impact
Maintain a nuanced view on progress, acknowledging that while growth has generally been good, new technologies like AGI could have unforeseen negative consequences. Reframe the perception of historical “geniuses” by understanding that modern intellectual talent is abundant but faces different innovation landscapes.
10. Mitigate New Addictions
Be aware that technological progress can create new forms of addiction and “traps” that exploit human psychology, such as social media and processed foods.
11. Societal Problem Mitigation
Recognize that societal problems caused by technology can often be mitigated through collective action and regulation, as seen with air pollution.
12. Focus AI/Bio Safety
Focus safety efforts on specific, foreseeably high-risk technologies like bioweapons and AI, rather than broadly restricting all innovation. Pursue both technical solutions and regulatory frameworks for AI safety, as they are complementary and mutually reinforcing.
13. Learn from Pandemics
Do not assume society will automatically learn from and effectively respond to major crises like pandemics without deliberate effort. Advocate for low-cost, high-impact interventions (e.g., improved air circulation) in public health responses.
14. Identify AI Research Threshold
Identify AI’s ability to autonomously conduct high-level AI research as a critical threshold for exponential progress and heightened risk, signaling a need for extreme caution.
15. Separate Responsibility, Outcome
Differentiate between responsible action and achieving a good outcome, recognizing that positive results can sometimes occur despite irresponsible approaches.
16. Prioritize Career Fulfillment
When giving career advice, prioritize what individuals genuinely have energy for and can excel at, rather than solely focusing on external impact metrics. Approach complex, high-stakes work like AI development with positive energy and high integrity, as personal well-being correlates with a reduced risk of inadvertently causing harm.
17. Strategic Impact Allocation
When evaluating impact, prioritize direct, comparable benefits (apples-to-apples) and seek to maximize them. Balance quantitative comparisons with intuitive diversification when comparing incommensurable benefits to avoid excessive noise and negative consequences.
18. Skepticism of Pure Philosophy
Use philosophical thought experiments to challenge intuitions, but prioritize your core heuristics and values over purely theoretical philosophical arguments in high-stakes decisions. The methodology of philosophy is often not robust enough to outweigh practical judgment.
19. Support Neglected High-Impacts
Direct attention and effort towards highly neglected but impactful areas such as AI safety, pandemic preparedness, animal welfare, and global poverty. These fields offer immense opportunities for doing good that are currently under-resourced.
20. Value Modern Social Science
Prioritize learning from contemporary social science for higher quality reasoning and understanding, noting its consistent improvement. Modern social science, particularly in causal inference, has become significantly more rigorous and reliable than in past decades.
5 Key Quotes
If you look at the whole trend and extrapolate it, you would project explosive growth. And there's been papers making that argument. You could also expect things to collapse.
Holden Karnofsky
I think when people imagine what the future is going to be like, they think about all they've ever known. But all they've ever known is a tiny fraction of human history.
Holden Karnofsky
It's like, I don't know, like humans are pretty different from each other. We're pretty different from ourselves or we're confusing or we're inconsistent. There's a lot of stuff we want. There doesn't seem to be any one thing that we strategically prioritize over everything else that varies by the human. It's a little hard to predict.
Holden Karnofsky
I feel like natural selection is a bit of a worst case for how you would train an AI.
Holden Karnofsky
I think philosophy is an extremely unimpressive field, in my opinion, the methodology just isn't very good. And there's not much reason to think we've like learned a lot from doing philosophy.
Holden Karnofsky