Beyond cognitive biases: improving judgment by reducing noise (with Daniel Kahneman)

1. Implement Mediating Assessment Protocol

Apply the mediating assessment protocol by defining important attributes for a decision, evaluating each attribute independently, and delaying the global judgment until all attributes have been assessed. This structured approach helps avoid premature conclusions and gathers more valuable information.

2. Employ Algorithms for Judgment

Use algorithms, even simple rules, for making judgments, especially in data-rich environments, because they are noise-free and consistently apply rules, often leading to more accurate outcomes than human judges.

3. Implement Noise Audits

Conduct noise audits by having multiple individuals in the same role make judgments on identical cases and then comparing their variability. This helps quantify and reduce noise in organizational decisions, even without knowing the true “bullseye.”

4. Average Independent Judgments

Gather independent judgments from multiple people on the same issue and average them to significantly reduce noise and improve accuracy, even if individual biases persist. This leverages the “wisdom of the crowd” effect to eliminate noise.

5. Maintain Independent Group Judgments

To maximize the benefits of group judgment, ensure individuals form and record their opinions independently before any group discussion. This prevents judgments from contaminating each other, which can otherwise reduce the noise-reduction effect and even lead to polarization.

6. Average Your Own Judgments

Improve your personal judgments by consciously generating multiple perspectives or judgments on a single issue, such as through a pre-mortem exercise, and then averaging them. This “crowd within” approach can reduce individual noise and improve accuracy.

7. Cultivate Disciplined Thinking

Adopt disciplined thinking by consciously applying decision hygiene rules and principles to important decisions, especially when you suspect you might be making a mistake. This structured approach can make your careful thinking more effective.

8. Apply Reference Class Forecasting

When making predictions, view the current situation as an instance of a broader class of similar past events, a technique called reference class forecasting. This method tends to reduce both noise and bias in your judgments.

9. Prioritize Organizational Improvement

Focus efforts on improving judgment and decision procedures within organizations, as this approach holds more promise for significant and lasting change than solely attempting to de-bias individuals.

10. Frame Judgment as Measurement

Conceive of judgment as a measurement operation, assigning a value on a scale, to apply the theory of measurement accuracy, which characterizes error in terms of bias and noise. This reframing helps in understanding and addressing errors in human judgment.

11. Acknowledge Noise in Judgment

Actively recognize and account for noise—the variability of errors—in judgments, as it is often neglected compared to bias, yet contributes equally to total error. Understanding noise helps identify inconsistencies in decisions.

12. Inspect Algorithms for Bias

Actively inspect the design and training data of algorithms to identify and mitigate potential biases, such as those stemming from disproportionate data for certain groups or biased input measures. These errors are detectable in principle and can be controlled.

13. Consider Algorithmic Caveats

Be cautious about using algorithms in rapidly changing environments where training data may not reflect new conditions, when humans possess critical, unquantifiable information, or when there’s a risk of feedback loops biasing training data. These situations may reduce algorithmic effectiveness.

14. Acknowledge Predictability Limits

Recognize that there are inherent limits to how predictable environments and life events are, even with advanced algorithms and extensive data. This understanding helps manage expectations and prevents overconfidence in forecasts.

15. Improve Research Practices

To enhance the replicability and scientific quality of social science research, adopt practices like using larger samples, pre-registering statistical analyses, and comprehensively reporting all procedural details. These measures help reduce self-deception and improve scientific rigor.

16. Recognize and Develop Ideas

Cultivate the ability to recognize when a “good idea” or a “glimmer of something new” spontaneously occurs, as this recognition and subsequent development are crucial for impactful work, rather than just deliberate searching.

17. Observe for Common Sense Insights

Cultivate keen observation of people, the world, and yourself, as many influential ideas stem from simple, common-sense insights that existing theories or disciplines may have overlooked. These insights can then be honed into scientific contributions.

18. Refine Ideas with Rigor

Transform common-sense ideas into precise, rigorous concepts, then experimentally test them and build a supporting theory. This process makes the ideas scientifically useful, especially when they challenge or expand existing knowledge.

19. Characterize Thinking Constructively

Approach the study of human judgment by constructively characterizing how people think, recognizing that errors are often predictable and systematic, rather than simply labeling behavior as “irrational.” This allows for a more productive understanding of cognition.

20. Apply Bayesian Evidence Framework

Utilize Bayesianism as a normative framework for evaluating evidence by considering how much more likely observed evidence is if a hypothesis is true compared to if it’s not. This provides a clear rule set for updating beliefs in light of new information.

21. Be Skeptical of Self-Improvement

Be realistic about the limited ability of individuals to greatly improve their own judgment simply by knowing about biases and noise, as personal experience suggests it’s difficult. Focus efforts more on improving organizational decision procedures, which have greater potential for impact.