Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

What changed This research investigates the distribution of learning gains across transformer layers during reinforcement learning (RL) post-training for large language models (LLMs). Traditionally, RL adaptation methods update all model parameters uniformly, operating under the implicit assumption that each layer contributes similarly to performance improvements. However, this study systematically examines the impact of training individual layers in isolation. The findings reveal a surprising concentration of RL gains within a small number of transformer layers, often just a single layer. This phenomenon was observed across seven different models, including variations within the Qwen3 and Qwen2.5 families, utilizing three distinct RL algorithms: GRPO, GiGPO, and Dr. GRPO. The experiments covered diverse task domains such as mathematical reasoning, code generation, and agentic decision-making.

A key metric introduced is 'layer contribution,' which quantifies the proportion of full RL improvement recoverable by training a single layer. Across all tested configurations, the study consistently found that a single transformer layer could achieve a substantial portion of the performance gains obtained from updating the entire model. In some instances, training just one layer even surpassed the results of full-parameter training.

Furthermore, the research identified a consistent structural pattern: the layers exhibiting the highest contribution to RL gains are typically located in the middle of the transformer stack. Layers positioned near the input and output of the model generally showed considerably less impact. The rankings of these high-contribution layers remained remarkably stable, showing strong correlations across different datasets, tasks, model families, and RL algorithms.

Why it matters for builders These findings have significant implications for developers working with LLMs. The current practice of full-parameter fine-tuning, especially with RL, can be computationally intensive and time-consuming. The discovery that a small subset of layers, potentially even a single one, can capture most of the performance benefits suggests a path toward much more efficient adaptation strategies. Builders could potentially achieve similar or even better results by focusing their computational resources on these critical layers, leading to faster iteration cycles and reduced costs for deploying and refining LLMs.

This efficiency gain could democratize access to advanced LLM capabilities, allowing smaller teams or individuals with limited resources to fine-tune models effectively. Understanding which layers are most sensitive to RL adaptation can also inform architectural choices and future model design, potentially leading to more parameter-efficient LLMs from the outset.

Practical impact The practical impact for developers lies in the potential for drastically reduced computational requirements for RL fine-tuning. Instead of updating millions or billions of parameters, future methods might involve identifying the few key layers responsible for learning specific tasks or behaviors. This could translate to:

* **Faster training times:** Significantly cutting down the hours or days required for RL post-training. * **Lower hardware costs:** Reducing the need for extensive GPU clusters. * **Easier experimentation:** Enabling quicker A/B testing of different RL strategies or hyperparameters. * **More accessible deployment:** Making advanced LLM customization feasible for a wider range of applications and organizations.

For instance, a developer aiming to improve an LLM's code generation capabilities might only need to fine-tune a specific set of middle layers, rather than the entire model, to achieve significant improvements. This targeted approach could also lead to smaller, more specialized models that retain core capabilities while excelling in specific domains.

Caveats and source limits The findings presented in this research are based on a study of specific models (Qwen3, Qwen2.5) and RL algorithms (GRPO, GiGPO, Dr. GRPO) across particular task domains. While the patterns observed were remarkably stable within these experiments, their generalizability to all LLMs, other RL algorithms, or different types of tasks has not been exhaustively demonstrated. The research paper introduces the concept of 'layer contribution' but does not provide a universally applicable method for identifying these layers a priori without performing some form of layer-wise analysis. Further research would be needed to validate these findings across a broader spectrum of models and training paradigms, and to develop practical, automated methods for identifying and updating only the most impactful layers.