Success can lead to a claim that can undermine the habits of mind that produced the success.
In war, everything is uncertain.
~ Helmuth von Moltke, Prussian field marshal (1800-1891)
We make decisions based on forecasts of people for whom we don’t have any track record of the accuracy of those predictions. Too many people are good at crafting compelling narratives and never have to go back to reconcile with their wrong forecasts.
Vague language and predictions are hard to measure. For example: “serious possibility” or “fair chance” — is it 20 or 80%? People don’t want to use numbers because they feel unnatural and not as expressive as language. “I would rather be a bookie than a poet.” Vague language is also easier to justify retrospectively, especially if the estimate is wrong.
People with the worst results in forecasting were those who stuck to an ideology, were hard to change their minds, and were overconfident.
The Good Judgment Project enlisted regular people—for example, retirees—who achieved significantly better results than the intelligence community, their resources, and algorithms.
Techniques and attitudes of great forecasters
- Triage. Concentrate on efforts that pay the most (get the most insights and so on) and that are achievable.
- Break seemingly intractable problems into multiple smaller tractable problems. Fermi method.
- Strike the right balance between the inside and outside view. Start with the baseline for that particular situation and estimate from there.
- Strike the right balance between overcorrecting and undercorrecting due to new evidence. “When facts change, I change my mind.”
- Look for the clashing causal forces in each problem. Look for opposing evidence or argument to avoid confirmation bias.
- Strive to distinguish as many degrees of doubt as possible. Go away from vague verbiage toward precise numbers. Kahneman warned about emotive responses to decisions and numbers in most people.
- Get comfortable with randomness instead of believing in fate or divine order. Believing in fate (“it was meant to happen”) is inversely correlated with forecasting accuracy (surveys of participants). Finding meaning helps with coping and resilience but not with forecasting.
- Time frame matters (most people are scope insensitive). “Will a regime fall in the next 24h” differs from “Will it fall in the next three months.” Bait and switch: people replace the time frame question with the one about the sustainability of the regime.
- Balance prudence and decisiveness. You need to listen and think but also act and take a side (don’t stay close to “maybe” or 50/50 chances). Especially critical for leaders.
- Look for errors behind your mistakes, but be aware of rear-view mirror hindsight biases. Don’t justify your failures, but own them too. Postmortem on both successes and failures.
- Team management and perspective taking, constructive confrontation. Teams were about 20% better forecasters than individuals. When a superforecaster was placed into a team, their predictions improved by 50%. More people can gather more diverse information. They were sharing analysis and judgment and getting feedback. Psychological safety was critical.
- Lifelong learners. Learning requires doing. One needs feedback. Deliberate practice. Growth mindset.
Viability of long-term forecasting
Book reference: Black Swan by Nassim Nicholas Taleb
It’s harder (or even impossible) to predict highly improbable events.
Every decade in the 20th century looked different than anticipated before. If you want to prepare for the long term, prepare for surprise and resilience. From Thoughts for the 2001 Quadrennial Defense Review (PDF) written in April 2001 by Linton Wells.
When forcasters approach the 5-year horizon in their forcasts, the accuracy approches that of random chance because there’s too much chaos in the world. Everything after five years is a flip of a coin.
Our minds crave certainty. We fill in the gaps with stories.