Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot
Systems that generate scientific research questions are evaluated today by expert scores, LLM-as-judge ratings, or curated case studies -- all subjective, none falsifiable. We formalize historical backtesting as an alternative: a system generates questions from a corpus frozen at a historical cutoff, the questions are frozen before any access to later literature, and a temporally isolated future…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
