DistIL: Reinforcement Learning from Rich Feedback

What changed

Researchers have introduced DistIL, a novel approach to reinforcement learning (RL) for reasoning models that moves beyond the standard Reinforcement Learning from Verifiable Rewards (RLVR) paradigm. RLVR typically relies on a single bit of feedback per response, indicating correctness, which offers limited credit assignment. DistIL, however, is designed to utilize richer feedback signals, including execution traces, tool outputs, expert corrections, and model self-evaluations. It is framed as a distributional variant of the classic imitation learning algorithm DAgger. The core of DistIL is a forward cross-entropy objective that allows for sequence-level gradient propagation, enabling rich credit assignment by propagating future expert-student disagreement back to earlier decisions. The authors demonstrate that prior RL methods using self-distillation objectives based on reverse KL or Jensen-Shannon divergences can fail to guarantee monotonic policy improvement, even when the expert feedback is superior. In contrast, DistIL's forward cross-entropy objective is shown to admit monotonic policy improvement and provides guarantees on regret. Furthermore, DistIL optimizes a lower bound on teacher-weighted likelihood of success, which is linked to improved Pass@N metrics. Empirically, DistIL has shown improvements over RLVR and other RL with self-distillation baselines across domains like scientific reasoning, coding, and solving complex mathematical problems. The accompanying code repository is available on GitHub, and a project website provides further details.

Theoretical Contributions

• Limitations of Existing Methods: The paper analyzes on-policy self-distillation objectives based on f-divergences, including reverse-KL and Jensen-Shannon. It proves that these methods do not generally guarantee monotonic policy improvement and that approximate gradients can lead to local credit assignment, potentially causing convergence to suboptimal policies.
• DistIL Algorithm: DistIL optimizes a forward cross-entropy loss between a feedback-conditioned teacher policy and the student policy on states visited by the student. It can accommodate black-box teachers and performs future-aware credit assignment.
• Theoretical Guarantees: DistIL's forward cross-entropy loss guarantees monotonic policy improvement, achieves sublinear regret, and maximizes a teacher-weighted lower bound on the expected log-likelihood of success.

Empirical Validation

• DistIL was evaluated on scientific reasoning, coding, and challenging mathematical reasoning tasks.
• Figure 1 in the paper compares DistIL with the SDPO algorithm (an RL with self-distillation method) on Qwen3-8B across four scientific reasoning domains (biology, chemistry, materials, physics). DistIL consistently achieved higher validation performance and demonstrated greater stability compared to SDPO, which exhibited more variability and occasional performance declines.

Why it matters for builders

DistIL offers a more sophisticated way for AI builders to train models, especially those involved in complex reasoning tasks. By moving beyond simple correct/incorrect feedback, developers can now leverage more nuanced and informative signals from experts or execution environments. This capability is crucial for developing AI systems that can perform intricate tasks in domains like scientific discovery, software development, and advanced mathematics, where intermediate steps and detailed feedback are vital for learning.

Practical impact

AI builders can explore implementing DistIL in their training pipelines to enhance the performance of reasoning models. The availability of a GitHub repository for DistIL allows for direct experimentation and integration. Developers should consider applying DistIL to tasks where rich feedback is obtainable, such as code generation with unit tests, scientific hypothesis generation with detailed critiques, or mathematical problem-solving with step-by-step verification. The theoretical guarantees of monotonic improvement and regret minimization suggest that DistIL could lead to more stable and predictable training outcomes.

Caveats and source limits

The primary source for this information is a research paper available on arXiv. While the paper presents theoretical guarantees and empirical results, it does not include details on specific model sizes used beyond mentioning Qwen3-8B in the context of experimental figures, nor does it provide pricing information or release dates for a production-ready version of the DistIL algorithm. The empirical results are based on the authors' experiments, and independent benchmarks are not yet available. The code is provided as a research artifact, and its readiness for large-scale production deployment would require further evaluation.