Information-Time Proximal Policy Optimization
RLVR has substantially improved the reasoning capabilities of LLMs. However, existing methods typically parameterize temporal progression in the Markov Decision Process by token-by-token generation, despite the highly non-uniform information flow along autoregressive trajectories. In this paper, we propose InfoPPO, which reparameterizes temporal progression using information density rather than…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
