Artificial intelligence now beats some of the best human forecasters
Crystal balls give way to LLMs
On September 5th, artificial intelligence outperformed human forecasters in the seasonal Metaculus Cup, a competition that gauges forecasters' accuracy across various events. In this contest, an AI emerged victorious, with other AIs securing second and fifth places, while humans claimed third and fourth. Hundreds of entrants predicted outcomes resolved by September, such as state or EU data-centre restrictions, hantavirus outbreak effects, and Brent crude cost.
Participants were scored based on how close their predictions were to the actual answers, with the closest forecasters accumulating the most points. The human participants shared a $5,000 prize, which amounted to less than the fifth-place AI's real-world gains. Through the use of prediction market investments, FutureSearch AI gained a 6% increase on its $100,000 portfolio since June.
Meanwhile, a Preseen developer turned $35 into $1.94 million in seven months, ranking sixth in Kalshi's history. AI forecasters differ from traditional ones, as they are not trained on specific data but are instead built using large language models (LLMs) that mimic human reasoning. This allows the AI bots to provide explicit reasoning behind their forecasts, which is useful for convincing decision-makers.
Modern forecasting has become more data-driven, making it easier to identify unreliable predictors. Research published in 2015 found that superforecasters could predict future events with the same reliability as those with a shorter time horizon. AI systems have now reached parity with superforecasters in forecasting accuracy. The success of AI forecasters can be attributed to the investments made by a few companies in scaling up large language models.
Some startups combine LLMs from different firms to investigate different aspects of forecasting questions, assemble data, and debate the correct response. Notably, a British startup called Mantic uses unique datasets inaccessible to publicly available AI models to enhance its AI's performance. AIs can also wipe their memory, allowing developers to test different instructions and past data, helping them learn and improve continuously.
While the exact reason for AIs' success over human forecasters remains unclear, it is evident that AI's processing power will increasingly collaborate with human expertise, allowing both to make more accurate predictions. Currently, human forecasters charge high fees and take ample time to provide forecasts, while AIs offer a faster, more affordable alternative.
Written by urgent.news from Hindustan Times - World News's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
