Representing emotive discourse in Ukrainian-English literary translation: A multi-method performance evaluation of Large Language Models, Neural Machine Translation and Computer-Assisted Translation tools

Abstract

The study examines the capacity of modern translation technologies to render Ukrainian literary texts into English. Lesya Ukrainka’s Letter to Serhii Merzhynskyi was chosen as the original text for translation analysis. It is a piece of emotive discourse marked by vivid imagery, nuanced stylistic features, expressive syntactic patterns and archaic vocabulary. Six translation services were tested. They included general-purpose Neural Machine Translation services, Computer-Assisted Translation tools and Large Language Models. Their output was evaluated using a three-step methodological framework. First, automatic evaluation was conducted using a Bilingual Evaluation Understudy (BLEU) metric to provide initial quantitative comparability across the systems’ output. Second, a qualitative analysis was undertaken through the concept of literariness, focusing on literature-specific features, aesthetic and stylistic peculiarities that distinguish literary texts from non-literary ones. In the final stage, human evaluation was employed, with five human annotators – native speakers with advanced linguistic proficiency, professional translators and scholars – ranking sentences to assess MT performance. The results of human evaluation and qualitative analysis revealed that the top-performing translation technologies were LLMs ChatGPT-5 and DeepSeek, which not only met a baseline level of translation adequacy but also consistently surpassed human translation in contextual and emotional sensitivity and overall naturalness and fluency. By contrast, automatic evaluation using the BLEU metric assigned the highest score to Google Translate output, highlighting the metric's limitations for literary text. Despite the notable efficiency of modern translation technologies, certain errors persist to varying degrees across all tested tools. These errors are connected with rendering imagery, handling syntactic constructions with long-range dependencies, translating pronouns, handling register mismatches, disrupting tone and other similar issues.