Skill Entropy Benchmark and RL for LLM Long-Horizon Reasoning

What changed

Researchers have introduced a novel approach to evaluate and train Large Language Models (LLMs) in long-horizon reasoning, focusing on their ability to switch between distinct skills. This capability is essential for tasks that require a sequence of different reasoning steps, such as performing a mathematical derivation and then using the result to plan a schedule. The paper formalizes these as "cross-skill long-horizon tasks." To address the lack of principled evaluation for skill switching, the authors introduce "Skill Entropy," a measure quantifying the difficulty of transitioning from one skill to another. This concept is applied to create "Skill^2-Bench," a new benchmark comprising cross-skill long-horizon tasks built upon 558 skills across 9 verifiable and open-ended domains. Each task in Skill^2-Bench is assigned a task-level skill entropy score and categorized into three difficulty levels. Evaluations of 8 frontier and 4 open-source models on this benchmark revealed a "skill-switching gap," where model accuracy decreases as the task-level skill entropy increases. Furthermore, models showed a performance drop of approximately 4-13% when using a skill within a cross-skill task compared to a single-skill question, even for skills they handle well in isolation. A dominant failure mode identified is the tendency for models to reuse the skill and answer modality from the previous step instead of switching to the skill required by the current step.

Beyond evaluation, the researchers propose "Skill-Entropy RL," a Reinforcement Learning framework that transforms skill entropy into a training signal. In this framework, the LLM predicts not only the answer at each step but also the specific skill employed to generate it. The reward function combines step-level correctness with a skill-entropy reward that measures the alignment between the model's predicted skill sequence and the ground truth skill sequence. This approach was applied to Qwen3-4B-Instruct and Qwen3-1.7B models, significantly improving their Skill^2-Bench scores from 34.4% to 68.4% and from 14.6% to 40.1%, respectively. These improvements surpassed competitive baselines. The Skill-Entropy RL pipeline is also adaptable to existing training data, such as OpenR1-Math, demonstrating the reusability of skill entropy as a training signal.

Why it matters for builders

This research provides builders with critical new tools for developing and assessing LLMs capable of complex, multi-step reasoning. The introduction of Skill Entropy and the Skill^2-Bench benchmark offers a standardized way to measure and identify weaknesses in how models handle skill transitions, a key bottleneck in current LLM performance on long-horizon tasks. The Skill-Entropy RL framework offers a concrete method to train models to overcome this limitation, potentially leading to more robust and capable AI agents and applications that require sophisticated sequential decision-making.

Practical impact

Builders can leverage the Skill^2-Bench benchmark to evaluate their models' skill-switching capabilities and identify specific areas for improvement. The Skill-Entropy RL framework, along with its associated code available on GitHub, provides a practical approach for fine-tuning existing models or training new ones to enhance their performance on cross-skill reasoning tasks. The adaptability of the training pipeline to off-the-shelf data like OpenR1-Math suggests that builders can integrate this technique into their existing training workflows to boost performance on tasks requiring sequential skill application.

Caveats and source limits

The Skill^2-Bench benchmark is built upon 558 skills across 9 domains, and while comprehensive, its coverage may not encompass all possible reasoning skills or domains relevant to every application. The performance improvements reported for Qwen3 models are specific to those architectures and training regimes; broader applicability across diverse LLMs will require further investigation. The research paper is currently available as a preprint on arXiv, and independent verification of the benchmark results and the efficacy of the Skill-Entropy RL framework by the wider research community is pending.