SimOPD: Relational Knowledge Transfer for On-Policy Distillation
A key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge. Supervision from token probabilities may underutilize internal knowledge, while direct hidden-state matching forces the student to shift to the teacher's feature space and complicates cross-architecture transfer. Inspired by contrastive learning's use of relationships between instances, we…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
