Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO
RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
