Small Language Models Compete with Frontier LLMs in Relation Extraction

What changed This research investigates the effectiveness of small language models (SLMs) in relation extraction (RE), a task where large language models (LLMs) have shown strong performance. The study challenges the notion that only massive LLMs can achieve state-of-the-art results by evaluating five SLMs ranging from 360 million to 3 billion parameters. These models were tested across various domain-composition regimes and tuning styles, with a focus on comparing their performance against zero-shot frontier LLMs and a discriminative RoBERTa baseline. The findings indicate that targeted task adaptation can enable compact models, even those with as few as 360 million parameters, to outperform general-purpose frontier LLMs when evaluated under specific protocols. For instance, a fine-tuned Qwen2.5-0.5B model achieved a general-domain positive-class micro-F1 score of 0.83, surpassing GPT-5.4 (0.69) and Claude Sonnet 4.6 (0.66) when used zero-shot. This performance gain is attributed to task adaptation rather than inherent generative decoding capabilities, as an in-domain RoBERTa baseline also outperformed the frontier models. On literary RE tasks, fine-tuned SLMs reached a score of 0.92 on the Biographical benchmark, compared to 0.83 for GPT-5.4, and an average of 0.833 versus 0.578 on a two-benchmark literary average. Further experiments, including a domain-adaptive pretraining case study, showed no significant gains over supervised fine-tuning, and a within-family scale comparison revealed only marginal improvements. The study concludes that when task-specific data is available, compact, task-adapted models offer an accurate, private, and hardware-efficient solution for RE.

Why it matters for builders For AI builders, this research offers a compelling case for reconsidering the use of SLMs. The ability of fine-tuned SLMs to rival or exceed the performance of much larger, often proprietary, LLMs on specific tasks like relation extraction is a game-changer. It means that developers can potentially build sophisticated AI applications that are deployable on consumer-grade hardware, such as a single GPU. This is particularly relevant for applications requiring low latency, offline processing, or enhanced data privacy, where sending data to external APIs is not feasible. The findings empower builders to create more accessible and cost-effective AI solutions without compromising on accuracy for targeted use cases.

Practical impact The practical implications of this research are substantial for developers working with relation extraction. The study suggests that instead of relying on large, resource-intensive LLMs, builders can opt for smaller, fine-tuned models that are significantly more efficient in terms of computational power and memory. This efficiency translates to lower operational costs and the possibility of deploying AI models on edge devices or in environments with limited infrastructure. For instance, a company needing to extract specific relationships from a large corpus of internal documents could fine-tune an SLM on their data, achieving high accuracy while maintaining data privacy and reducing processing time. The research also highlights that the gains are primarily from effective task adaptation, suggesting that investing in high-quality, domain-specific datasets and fine-tuning strategies can yield superior results compared to simply using the largest available general-purpose models.

Caveats and source limits The findings presented in this research are based on a specific set of benchmarks and evaluation protocols. While the study demonstrates the potential of fine-tuned SLMs, it is important to note that the comparison with frontier LLMs was conducted under a zero-shot setting for the latter. The performance gains for SLMs are heavily dependent on task-specific adaptation and the availability of relevant training data. The research does not imply that SLMs are intrinsically superior to LLMs in all aspects or for all tasks. Furthermore, the study focuses on relation extraction; its generalizability to other natural language processing tasks may vary. The specific models and configurations tested represent a snapshot of current capabilities, and the landscape of both SLMs and LLMs is rapidly evolving. The source is a pre-print on arXiv, meaning it has not yet undergone formal peer review, which is a standard part of the scientific publication process.