Does Forecasting Have Room At The Top?

The concept of "superforecasting" refers to the ability to predict future events with a high degree of accuracy, covering a wide range of scenarios from election outcomes to the discovery of key technologies. This field has evolved from a niche academic study to a lucrative industry of prediction markets. In recent times, AI-powered superforecasters have approached the precision of top human forecasters, and their performance is showing an upward trajectory.

In the near future, we might witness one of two outcomes: either humans have reached a certain limit in predicting the predictability of world events, in which case AI models will stabilize at or just above the human level, or the trend will continue unabated, with AI surpassing even the top human forecasters. Daniel Reeves, a proponent of the former scenario, presents a study he coauthored in 2010 which suggests that prediction markets were only able to outperform basic statistical models by 3-6%.

This study focused on questions related to sports games and movie box office receipts. Reeves argues that statistical models are nearing their optimum potential, and the remainder of the prediction can be attributed to aleatoric uncertainty, which is the inherent unpredictability of chaotic systems. Despite Reeves' findings, the author remains skeptical, holding a 70-30 expectation for the second scenario.

The key difference between the author and Reeves lies in the interpretation of the prediction markets' performance. While Reeves concedes that the prediction markets have significantly improved upon previous statistical models, the author suggests that the improvement is marginal relative to the amount of information that is actually being incorporated, such as team specifics and their overall performance in sports.