Trust Region On-Policy Distillation
English summary
Trust Region On-Policy Distillation (TrOPD) is proposed to enhance on-policy distillation for large language models by mitigating instability caused by distribution mismatch between teacher and student. The method integrates trust region constraints, outlier estimation for token-level credit assignment, and off-policy guidance to stabilize policy gradients. Experiments show TrOPD outperforms existing on-policy distillation baselines across mathematical reasoning, code generation, and general-domain benchmarks.
Chinese summary
提出信任区域在策略蒸馏（TrOPD），针对大语言模型在策略蒸馏中因师生分布差异导致的策略梯度不稳定问题。TrOPD引入信任区域约束、离群点估计进行令牌级信用分配，并结合离策略指导增强优化稳定性。在数学推理、代码生成和通用领域基准测试中，TrOPD均显著超越现有在策略蒸馏基线方法。
Key points
TrOPD addresses instability in on-policy distillation from large teacher-student distribution gaps.
TrOPD解决了因教师与学生输出分布差异大而导致的在策略蒸馏不稳定问题。
It leverages trust region on-policy learning, outlier estimation for token-level credit assignment, and off-policy guidance.
方法利用信任区域在策略学习、离群估计进行令牌级信用分配，以及离策略指导。
TrOPD achieves superior results over baselines on math reasoning, code generation, and general tasks.
TrOPD在数学推理、代码生成和通用任务上取得了优于基线的性能。