Okie-dokie, Casey Handmer edition
I’ve been prototyping an automated peer review project. With current models you get human level peer review for about $0.10 of tokens. For $5 you can rederive every result, check every statistical inference, and audit every paper cited in both directions. There are about 100m papers in the scientific corpus. Most are not important. Many […] The post Okie-dokie, Casey Handmer edition appeared…
Casey Handmer is working on an automated peer review project using current models, which can provide human-level review for approximately $0.10 in tokens. By investing $5, one could rederive every result, verify all statistical inferences, and audit every paper cited in both directions. However, there are around 100 million papers in the scientific corpus, with most of them being unimportant or poorly executed.
Due to structural limitations, correcting published work, even for typos, is nearly impossible. For $500 million, it would be possible to thoroughly audit every paper ever written, identifying p-hacked results, plagiarists, fabrications, and citation clubs. As Handmer cannot afford this amount, he is focusing on a tree approach, targeting the 1% of papers that hold 99% of the signal.
He can find 90% of the nonsense for a few hundred dollars of tokens. The biggest surprises may be revealed in this process. The details can be found <here>. This post appeared on Marginal REVOLUTION.
Written by urgent.news from Marginal Revolution's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
