On-policy distillation asks a teacher to score the states that a student actually visits. In verifiable reasoning, each student response also receives an outcome label. These signals can conflict: a complete answer is correct, yet the teacher assigns one of its sampled tokens lower probability than the behavior policy that produced it.On-policy 蒸馏让教师评估学生实际到达的状态。在可验证推理中,每条学生回答还会得到结果标签。两种信号可能冲突:整条解答是正确的,但教师给其中一个采样 token 的概率低于生成它的行为策略。

At such a position, a standard teacher target gives that token a negative distillation advantage. An update can push against a verified solution. SP-OPD treats verifier acceptance as information about the target itself: on a correct trajectory, the sampled token must receive at least its behavior-policy probability.在这个位置,普通教师目标会给该 token 一个负的蒸馏优势,更新方向可能与已验证的正确解答相反。SP-OPD 把验证器的认可转成目标分布的约束:正确轨迹中的采样 token,目标概率不能低于其行为策略概率。

Project the teacher onto a feasible target将教师分布投影到可行目标

The method chooses the distribution closest to the teacher under KL divergence while satisfying that constraint. If the teacher already meets it, the target stays the same. At a conflict, projection lifts the sampled token to the constraint boundary and rescales the other tokens proportionally, preserving their relative probabilities.方法在满足约束的分布中,选择与教师分布 KL 距离最近的目标。教师原本满足约束时,目标保持不变;发生冲突时,投影只把采样 token 提到约束边界,并按比例重分配其余 token 的概率,保留它们之间的相对关系。

The outcome label applies to a whole response. It certifies the sampled correct trajectory, without naming an individual token responsible for an incorrect one. SP-OPD therefore applies the support constraint to correct trajectories and keeps the teacher target on incorrect trajectories.结果标签作用于整条回答。它确认采样轨迹正确,却不会指出错误回答究竟错在哪个 token。因此 SP-OPD 对正确轨迹施加支持约束,对错误轨迹沿用教师目标。

Carry one target through both loss terms让两项损失使用同一目标

SP-OPD uses the projected target in both the clipped reverse-KL advantage and the forward-KL term. This matters because fixing only a scalar advantage would leave the forward-KL target unchanged. Its response-balanced reduction also averages token losses within each response before averaging responses, matching the verifier's response-level label.SP-OPD 在截断的 reverse-KL 优势与 forward-KL 项中都使用投影后的目标。若只修正一个标量优势,forward-KL 仍会沿原教师分布更新。方法还先在每条回答内平均 token 损失、再平均回答,与验证器的整条回答标签保持一致。