On‑policy distillation works with limited additional data

AutoDAgger, a novel distillation technique, enables compact policies to inherit the skills of large models using only minimal additional data. By integrating the teacher's corrective actions directly into the student's learning loop, AutoDAgger avoids the resource-intensive data collection phase that has traditionally been a bottleneck in distillation pipelines.

Previous methods, such as pure behavior cloning and offline distillation, still required human oversight and a significant increase in dataset size, making scaling down high-capacity policies to edge devices challenging. In contrast, AutoDAgger gathers recovery data more than twice as quickly as its predecessor, HG-DAGGER, eliminating the need for human monitoring entirely.

With just 30 episodes of on-policy recovery, the student policy demonstrates remarkable improvement, jumping from a baseline score of 27.8 to 63.9, and reaching 77.8 after 90 episodes. This efficiency suggests that a smaller dataset can achieve performance levels comparable to those obtained with larger offline collections. The autonomous reward model monitors task progress and triggers teacher takeovers only when necessary, transforming a manual handoff into a rapid self-service pipeline.

Experiments on three manipulation tasks show that TILDE, the distillation method, achieves scores of 80.6, 63.9, and 69.2, outperforming all evaluated baselines.

While the study focuses on three specific tasks and assumes a reliable reward model to signal failure, it demonstrates that on-policy distillation can transfer near-teacher capability without extensive pretraining. The results align with those of original embodied foundation models, confirming that on-policy distillation can transfer close to teacher capability without exhaustive training.

However, the study leaves open whether similar speedups and performance gains can be achieved in more complex or less well-instrumented domains. The robustness of AutoDAgger in the face of sparse teacher interventions and noisy reward signals remains an open question for future research. As autonomous recovery replaces human-in-the-loop data collection, the benchmarks currently requiring tens of thousands of trajectories could be re-run with AutoDAgger, expecting comparable scores at a fraction of the data cost.

This shift could make policy distillation a practical solution for on-device deployment across various robotics applications.