I expected Jev to remember 2025. It doesn't know Trump was inaugurated
I expected Jev to have learned the outcomes of old prediction markets. It hasn't. Asked "Will Donald Trump be inaugurated?" about the January 2025 market, Jev says 0.70. Asked whether the Fed raised rates in January 2025, it says 0.30, the same number it gives almost everything. Across 17,036 Polymarket markets resolved up to 20 months before its release, Jev's ability to rank Yes from No (AUC…
The author expected a language model named Jev to be able to recall significant events from 2025, such as Donald Trump's inauguration. However, Jev only correctly answered one out of 17,036 markets related to the year 2025, while achieving a 0.30 probability for all other markets. The author conducted tests on different months leading up to and after the release of Jev, spanning from January 2025 to September 2026.
The results showed that Jev's performance was similar across all months, with a 0.68 accuracy rate, which is equivalent to random guessing. The author concluded that Jev does not possess the ability to remember events from 2025, contradicting the initial assumption.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.