Ben Recht Argues Uncertainty Quantification is Narrowly Defined

What changed

In a blog post titled "Unimaginative Uncertainty," Ben Recht, a professor at UC Berkeley, critiques the prevailing methods of quantifying uncertainty in forecasts. He argues that the field is overly reliant on error bars and prediction intervals, which he describes as a "woefully narrow" technocratic imagination. Recht points out that these methods often assume Gaussian distributions and lack creativity in their construction, typically involving estimating variance and setting bounds. He highlights that even when using more complex methods like Monte Carlo simulations or analyzing past errors, the underlying assumptions and modeling decisions make the resulting probabilities difficult to verify. Recht suggests that forecasters often use probabilistic predictions to avoid being definitively wrong, rather than for genuine decision-making.

Why it matters for builders

Recht's critique is significant because it challenges the fundamental assumptions behind many forecasting models used in AI and economics. His argument that current methods for quantifying uncertainty are limited implies that the confidence placed in these forecasts may be misplaced. For builders, this means re-evaluating how uncertainty is communicated and understood, and potentially exploring alternative methods that offer a more holistic view of potential outcomes and model limitations.

Practical impact

Recht's core argument is that the current technocratic approach to uncertainty quantification is insufficient. He advocates for a more imaginative approach that goes beyond simple error bars. This could involve developing methods that better capture the complexity of uncertainty, allowing for more robust preparation for when forecasts inevitably miss the mark. Builders should consider that the probabilistic statements from models might not fully represent the true uncertainty, and that a "good engineer" might need to multiply their error bars to account for unquantified uncertainties.

Caveats and source limits

Ben Recht's post is a personal blog entry reflecting his views on forecasting and uncertainty quantification. The claims made are his opinions and analyses based on his academic perspective. While he references Charles Manski's work and examples from weather forecasting and macroeconomic planning, the post does not present independently verified data or results. The primary limitation is that Recht's critique focuses on the *methods* of uncertainty quantification, suggesting that the *quantification itself* is often flawed or incomplete, rather than refuting specific numerical forecasts.