Small Language Models Outperform Frontier LLMs in Relation Extraction

What changed

This research investigates the potential of small language models (SLMs), ranging from 360 million to 3 billion parameters, to match or exceed the performance of large language models (LLMs) on relation extraction (RE) tasks. The study evaluated five different SLMs across 30 configurations, combining three domain-composition regimes and two prompt-conditioned tuning styles. These adapted SLMs were compared against zero-shot frontier LLMs, including GPT-5.4 and Claude Sonnet 4.6, as well as a discriminative RoBERTa baseline. Across nine benchmarks, the best performing sub-billion parameter model, Qwen2.5-0.5B, fine-tuned on pooled general-domain data, achieved a positive-class micro-F1 score of 0.83 for general-domain RE. This surpasses the zero-shot performance of GPT-5.4 (0.69) and Claude Sonnet 4.6 (0.66). The study highlights that this performance gain is attributed to targeted task adaptation rather than inherent SLM superiority. An in-domain RoBERTa baseline also outperformed the frontier LLMs, further supporting the notion that task adaptation is key. For literary RE, fine-tuned SLMs reached an F1 score of 0.92 on the Biographical benchmark, compared to 0.83 for GPT-5.4. On a two-benchmark literary average, tuned SLMs achieved 0.833 versus 0.578 for GPT-5.4. A case study on domain-adaptive pretraining (DAPT) showed no significant improvement over supervised fine-tuning for literary RE, suggesting that supervised task adaptation is the primary driver of performance gains. Within-family comparisons also indicated only marginal improvements from scaling.

Why it matters for builders

For AI builders, this research presents a compelling case for reconsidering the use of smaller, more manageable models. The findings suggest that with appropriate fine-tuning and task-specific data, SLMs can deliver performance competitive with, and in some cases superior to, much larger and more resource-intensive frontier LLMs. This is particularly relevant for developers working under constraints related to computational power, budget, or data privacy. The ability to deploy accurate RE capabilities on a single consumer GPU, as demonstrated by the 4-bit models evaluated, democratizes access to advanced NLP functionalities. It enables the creation of applications that can run locally, enhancing user privacy and reducing operational costs associated with API calls to proprietary models.

Practical impact

AI builders can leverage these findings by exploring fine-tuning strategies for available SLMs on their specific relation extraction tasks. The research indicates that focusing on supervised fine-tuning with domain-specific data can yield substantial performance improvements. Developers should consider evaluating models like Qwen2.5-0.5B for general-domain RE and similar-sized models for literary text, especially when resource efficiency and privacy are paramount. The study provides concrete benchmarks and methodologies for adapting models, suggesting that developers can achieve high accuracy without necessarily adopting the largest available LLMs. This approach allows for more flexible deployment scenarios, including edge devices and private cloud environments.

Caveats and source limits

The study's findings are based on a specific set of benchmarks and evaluation protocols, and the authors note that the comparison does not imply intrinsic superiority of SLMs but rather the effectiveness of task adaptation. The research did not explore out-of-domain transfer or cross-domain generalization capabilities for the specialist models. While domain-adaptive pretraining was investigated, its practical gain over supervised fine-tuning was found to be minimal in the tested literary RE scenario. The performance of frontier LLMs was evaluated in a zero-shot setting, which may not represent their full potential when fine-tuned. The study focuses on relation extraction and may not generalize to other NLP tasks. The specific versions of frontier LLMs (GPT-5.4, Claude Sonnet 4.6) are hypothetical or internal designations, and their exact capabilities and availability are not detailed in the provided excerpt.