PAT: A RAG-Based System for Whole-Document Translation

What changed

This paper explores the potential of large language models (LLMs) to transcend the conventional sentence-by-sentence approach in automatic translation systems. Traditional tools, including Computer-Assisted Translation (CAT) and Machine Translation (MT), have largely operated under this paradigm. The research introduces PAT (Pragmatic Auto-Translator), a system built upon Retrieval-Augmented Generation (RAG). PAT integrates user-defined specifications with contextual information retrieved from a comparable corpus of longform texts. This corpus consists of authentic documents in U.S. English and Latin American Spanish. The system passes retrieved examples at paragraph, section, and document levels to an LLM, enabling whole-document generation. The primary objective is to produce draft translations that are reformulated to align with the specific context of the Spanish-speaking audience, accounting for differences in discourse organization, rhetorical style, and pragmatic norms compared to English.

An evaluation was conducted on six automatic translations of essays concerning generative AI. These translations were performed from U.S. English into Latin American and Mexican Spanish across three distinct projects. A customized Multidimensional Quality Metrics (MQM) typology was employed, with assessments made by two trained evaluators. The results indicated that a basic prompt yielded no significant reformulation. However, when specifications and corpus-informed translations were utilized, they sometimes resulted in substantial reformulation. While these reformulations were not always effective, the study concludes that LLMs can indeed be guided toward reformulation and away from the sentence-by-sentence paradigm. Nevertheless, further research is necessary to enhance the efficacy of these reformulations. The paper also delves into considerations for designing automatic translation systems, constructing relevant corpora, and methodologies for evaluating translation quality.

Why it matters for builders

PAT's approach offers a significant advancement for AI builders working on translation technologies. By moving beyond sentence-level processing, it addresses a key limitation in current MT systems, which often fail to capture the broader context and cultural nuances essential for high-quality translation. The RAG-based architecture, combined with corpus-informed retrieval, provides a framework for developing more sophisticated translation tools. This is particularly relevant for developers aiming to create applications that require not just linguistic accuracy but also cultural appropriateness, such as in global marketing, legal document translation, or cross-cultural academic research.

Furthermore, the research highlights the importance of corpus design and quality evaluation in achieving better translation outcomes. Builders can learn from PAT's methodology to construct more effective corpora and implement robust evaluation strategies. This focus on context and pragmatics can lead to the development of LLM-powered translation services that are more adaptable and performant in real-world scenarios, ultimately enhancing user experience and the utility of AI-driven translation.

Practical impact

The practical impact of PAT lies in its potential to improve the quality and contextual relevance of machine-translated content. For professional translators, PAT could serve as a powerful tool to generate initial drafts that are already more attuned to the target audience's linguistic and cultural expectations. This could significantly reduce post-editing time and effort. For businesses operating globally, more accurate and culturally sensitive translations can lead to better customer engagement, reduced misunderstandings, and more effective communication across different markets.

The system's ability to incorporate user specifications alongside corpus context allows for a degree of customization, enabling translations to be tailored to specific project requirements or industry jargon. This flexibility is crucial for specialized translation tasks where generic MT often falls short. The research also contributes to the broader understanding of how LLMs can be applied to complex linguistic tasks, paving the way for future innovations in natural language processing and cross-lingual communication technologies.

Caveats and source limits

The research presented in this paper is based on a single study evaluating six translations of essays on generative AI. While the results suggest that LLMs can be moved toward whole-document translation and reformulation, the effectiveness of these reformulations was not always consistent. The study acknowledges that more work is needed to improve the efficacy of these context-aware translations. The corpus used was specific to U.S. English and Latin American Spanish, and the evaluation was conducted by two trained evaluators using a customized MQM typology. The findings may not be directly generalizable to other language pairs, text types, or evaluation methodologies. The paper itself is a research paper, and PAT is presented as a system under development, not a production-ready tool. Further validation and refinement are implied as necessary steps.