This will make you a better decision-maker | Annie Duke (author of “Thinking in Bets” and “Quit,” former pro poker player)
Guest Annie Duke, author of "Thinking in Bets" and "Quit," discusses lessons from Daniel Kahneman, frameworks for improving decision quality in companies, and how to shorten feedback loops. She also shares insights on effective meeting structures, parenting, and the power of knowing when to quit.
Deep Dive Analysis
9 Topic Outline
Lessons Learned from Daniel Kahneman
Parenting Advice and Decision-Making Tools
The Extent of Improvement Possible in Decision-Making
Independent Brainstorming for Better Group Decisions
Ensuring People Feel Heard in Decision Discussions
Reframing Long Feedback Loops in Decision-Making
Improving Investment Decision-Making at First Round Capital
Effective Use of Pre-Mortems and Kill Criteria
The Challenges of Quitting and Knowing When to Walk Away
6 Key Concepts
Adversarial Collaboration
A method pioneered by Daniel Kahneman where researchers who disagree on a topic collaborate to design a study that could resolve their differences. This approach emphasizes open-mindedness and a willingness to be proven wrong in pursuit of truth.
Mental Time Travel
A decision-making tool that involves projecting oneself into the future to gain perspective on current problems or choices. By imagining how one might feel about a decision years later, it helps to reduce the 'focusing effect' of immediate emotions and importance.
Nominal Group Technique
A method for group decision-making that involves individuals working independently and asynchronously to generate ideas or judgments before coming together for discussion. This prevents biases like groupthink, influence from dominant voices, and premature convergence of opinions.
Curiosity vs. Coercion
In group discussions, a 'curiosity' model encourages participants to convey information and explain their beliefs without trying to convince others they are right. This contrasts with a 'coercive' model where the goal is alignment or agreement, which can suppress dissenting opinions and lead to less robust decisions.
Kill Criteria
Specific, pre-defined signals or thresholds that, if observed, indicate it's time to pivot or stop a project, investment, or course of action. These are established during a pre-mortem to combat biases like sunk cost and prevent continuing with failing endeavors.
Sunk Cost
The psychological tendency to continue an endeavor because of the time, money, or effort already invested, even if continuing is not rational. This bias makes it difficult to quit, as people fear 'wasting' past investments, rather than focusing on future costs and benefits.
8 Questions Answered
Kahneman was incredibly humble, open-minded, and curious, always seeking to understand why he might be wrong and engaging in adversarial collaborations. He prioritized learning and human connection over defending his past work, even admitting when studies didn't replicate.
Two key frameworks are 'mental time travel' to gain perspective on current issues by imagining future feelings, and using the word 'nevertheless' to acknowledge a child's feelings while still enforcing a decision, fostering a balance between being heard and following through.
Significant improvement is possible, as demonstrated by studies showing a 50% to 65% hit rate increase in hiring when using structured processes. However, this improvement hinges on actually implementing decision-making tools, which many people resist due to overconfidence in intuition.
The most impactful change is to ensure that opinions and judgments are gathered independently and asynchronously before any group discussion. This 'nominal group' approach prevents biases like groupthink and ensures all voices are heard without coercion.
Leaders should facilitate discussions by reflecting back what they hear from each person, clarifying understanding without expressing personal agreement or disagreement. This process ensures everyone feels their perspective was fully understood and considered, even if the final decision differs.
There's no such thing as a truly 'long' feedback loop; it's a choice. To shorten it, identify and track necessary but not sufficient conditions or highly correlated signals that must occur for the desired long-term outcome. This allows for earlier learning and adjustments.
The best use of a pre-mortem is not just to identify potential failures, but to establish 'kill criteria' – specific signals that, if observed, will trigger a pre-committed action to pivot or stop the project. This helps combat sunk cost bias and ensures rational action when things go wrong.
People struggle to quit due to sunk cost bias, endowment effect (overvaluing what they own), and concerns about identity or appearing to fail. By the time someone is actively thinking about quitting, it's often already past the optimal time, as they tend to wait until there's no other choice.
28 Actionable Insights
1. Make Implicit Assumptions Explicit
Take what’s implicit in your thinking and make it explicit to improve decision quality and identify when your intuition might be wrong.
2. Brainstorm Ideas Independently
Before group discussions, have individuals independently generate ideas, judgments, or forecasts with rationales, without seeing others’ input, to prevent cross-influence and capture a wider range of opinions.
3. Focus Meetings on Discussion
Restrict meetings solely to the discussion of ideas, ensuring that discovery (independent idea generation) and decision-making happen outside the group setting for greater effectiveness.
4. Make Decisions Independently
After group discussion, make the final decision outside of a collective meeting, either through a single designated decision-maker or via independent private voting, to avoid groupthink and social pressure.
5. Pre-Mortem with Kill Criteria
Conduct pre-mortems to identify potential failure signals for a project, then pre-commit to specific actions (kill criteria) you will take if those signals appear, helping to overcome sunk cost bias and enabling timely pivots or stops.
6. Shorten Long Feedback Loops
Identify and track intermediate signals or necessary conditions that are correlated with your desired long-term outcome, allowing you to get feedback much faster and improve decision-making without waiting years for the ultimate result.
7. Structure Unstructured Decisions
For complex decisions like hiring, break down implicit criteria into explicit components, create a decision rubric, and use structured processes (e.g., structured interviews) to drastically improve decision accuracy.
8. Employ ‘Nevertheless’ in Leadership
Acknowledge and validate others’ input and feelings (e.g., ‘I hear you’), but then clearly state the decision and the path forward (‘Nevertheless, this is what’s going to happen’) to balance empathy with decisive action.
9. Reject ‘Alignment’ as a Goal
Stop aiming for ‘alignment’ or unanimous agreement in group decisions, as it’s often unrealistic and can lead to coercive environments; instead, accept natural disagreement and focus on conveying information.
10. Convey, Don’t Convince
In discussions, aim to convey your ideas and the reasoning behind them, rather than trying to convince others you are right, to foster a culture of curiosity and open exchange of information.
11. Clarify and Reflect Input
When facilitating discussions, actively listen and reflect back what you heard others say to confirm understanding, without offering your own opinion, ensuring everyone feels heard and understood.
12. Use Structured Evaluation Rubrics
Create structured rubrics with defined criteria and precise rating scales for evaluating complex decisions (e.g., investments), ensuring consistency, reducing bias, and allowing for quantitative analysis of decision components.
13. Establish Shared Definitions
Ensure all participants in a decision-making process have a shared and explicit understanding of the terms and criteria being used (e.g., ‘market quality’) to prevent miscommunication and ensure consistent judgments.
14. Explicitly Forecast Probabilities
For long-term decisions, explicitly forecast the probability of key intermediate outcomes (e.g., a company funding at Series A) to make implicit predictions explicit, allowing for tracking of accuracy and feedback.
15. Track and Feedback Accuracy
Systematically track the accuracy of individual and team predictions and ratings against actual outcomes, then provide this feedback to decision-makers to foster continuous learning and improvement in specific areas.
16. Data-Driven Rubric Optimization
Continuously refine and optimize decision rubrics by analyzing data to determine which factors are truly predictive and which are not, ensuring the rubric focuses on the most impactful criteria.
17. Verify Intuition’s Predictiveness
Critically examine whether strongly held intuitions or emphasized factors are actually predictive of desired outcomes, as intuition can be both right and wrong, and testing it helps identify and correct non-predictive biases.
18. Assume Late Quitting
When considering quitting a project, job, or relationship, operate under the assumption that you’ve likely already waited too long, as strong human biases often cause people to delay quitting until it’s no longer a choice.
19. Apply ‘Start Today’ Rule
Evaluate ongoing commitments by asking yourself if you would choose to start them today, given all current information, to overcome the sunk cost fallacy and identify true waste.
20. Account for Opportunity Cost
When deciding whether to continue or quit, explicitly consider the opportunity cost – what other valuable endeavors you could be pursuing if you stopped the current one – to highlight hidden costs and make more rational decisions.
21. Cultivate Open-Mindedness Mindset
Actively seek out reasons why your current beliefs might be wrong and be willing to change your mind based on new information, as exemplified by Daniel Kahneman, to foster curiosity and uncover new insights.
22. Engage in Adversarial Collaboration
Partner with individuals who hold opposing viewpoints to jointly design studies or experiments that can resolve disagreements, objectively seeking truth rather than tearing each other down.
23. Gain Perspective on Importance
Remind yourself that current issues or decisions often feel more important in the moment than they truly are in the long run, helping to reduce emotional intensity and allow for a more rational assessment.
24. On-the-Fly Independent Brainstorming
If new ideas or challenges arise spontaneously in a meeting, pause the group discussion and ask everyone to independently write down their thoughts or estimates to prevent immediate groupthink and capture diverse perspectives.
25. Gain Edge with Frameworks
Actively apply decision-making frameworks and strategies, even if they are unnatural or difficult, to gain a significant competitive edge over those who know about these methods but fail to implement them.
26. Ensure Kids Feel Loved
Prioritize making sure your children deeply and unequivocally know that you love them, as this foundational love is the most important basis for their well-being, dwarfing other parenting styles.
27. Don’t Sweat Parenting Mistakes
Accept that mistakes will happen in parenting (e.g., dropping a baby), and don’t beat yourself up over them, as babies are resilient and not every small mistake will have a lasting impression.
28. Teach Kids Basic Manners
Focus on teaching your children fundamental manners like saying ‘please’ and ’thank you,’ as this is an achievable and valuable parental contribution, even if you have little influence on their core personality.
6 Key Quotes
Nothing is as important as it seems when you're thinking about it.
Daniel Kahneman (quoted by Annie Duke)
The only thing that's ever supposed to happen in a meeting is the discussion part.
Annie Duke
There is no such thing as a long feedback loop. You can make a decision about how long the feedback loop is. That is your choice to live in a long feedback loop. And you can choose to shorten the feedback loop.
Annie Duke
It's so incredibly necessary in improving decision quality to take what's implicit and make it explicit.
Annie Duke
Most people won't quit until it actually isn't a decision.
Richard Thaler (quoted by Annie Duke)
Waste is a prospective problem, not a retrospective one.
Annie Duke
2 Protocols
Effective Group Decision-Making Meeting Structure
Annie Duke- **Discover (Independently)**: Before the meeting, ask individuals to independently provide their opinions, judgments, forecasts, or brainstormed ideas. Ensure they submit responses privately (e.g., via a form) to prevent cross-influence.
- **Share & Discuss (In Meeting)**: Distribute all collected opinions to the group for review before the meeting. In the meeting, focus discussion on areas of disagreement, with a facilitator reflecting back what each person says to ensure they feel heard, without expressing personal agreement or disagreement.
- **Decide (Independently or by DRI)**: The final decision should not be made coercively in the group meeting. Ideally, a single designated responsible individual (DRI) makes the decision after hearing all input, or a private voting mechanism is used. Acknowledge that full alignment/agreement is not the goal, but rather understanding the full spread of opinions.
Pre-Mortem with Kill Criteria Protocol
Annie Duke- **Pre-Mortem Exercise**: Imagine the project or endeavor has failed (e.g., six months later, the deal is dead). Look back and identify early signals that indicated things were going wrong.
- **Establish Kill Criteria**: For each identified signal, define it as a 'kill criterion' – a specific, observable event or data point that would indicate trouble.
- **Pre-Commit to Actions**: For each kill criterion, commit to a specific, pre-determined action to take if that signal is observed (e.g., kill the deal, ask a direct question, offer executive alignment). This prevents inaction due to biases once the project is underway.