Prompting for Precision: Assessment of Translation Quality via Role-Based (Persona) Prompting in the Medical Texts Domain

Abstract

In both translation literature and the industry, the advancements in artificial intelligence (AI)-driven translation activities have led to a rapidly growing interest in prompt design for large language models (LLMs). Nevertheless, there is still a lot to unfold in terms of the impact of assigning specific roles or ‘personas’ within these prompts, particularly within the context of medical and pharmaceutical texts. This paper intends to respond to this gap by assessing the impact of employing role-based (persona) prompts when engaged in LLMs. The experiment evaluates the translation quality of outputs generated by role-prompted LLMs against two different source texts containing pharmaceutical content. The outputs were systematically compared to those generated by zero-shot prompted models, alongside the outputs obtained from a conventional neural machine translation (NMT) system: Google Translate. To evaluate the machine translation (MT) outputs, the study utilizes quantitative metrics such as BLEU (Bilingual Evaluation Understudy) and COMET (Crosslingual Optimized Metric for Evaluation of Translation). In parallel, a qualitative approach was employed by conducting a Translation Quality Evaluation (TQE) using a customized error-typology tool inspired by the Multidimensional Quality Metrics (MQM) framework. Consequently, the experiment was carried out through both quantitative and qualitative methodology, as a hybrid approach is essential for a solid quality assessment. Finally, the study presents comprehensive discussions concerning widely debated concepts such as domain-specific assessment and the potential for human–machine collaboration.

Keywords

• role-based prompting
• large language models
• translation quality evaluation
• medical texts
• human–machine interaction

References

1. Babar, Zia. “The Evolution of Prompt Engineering.” 2024. Medium, September 30. https://medium.com/@zbabar/the-evolution-of-prompt-engineering-3f67d2607473.
2. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2016. “Neural Machine Translation by Jointly Learning to Align and Translate.” arXiv:1409.0473v7. doi:10.48550/arXiv.1409.0473.
3. Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” arXiv:2005.14165v4. doi:10.48550/arXiv.2005.14165.
4. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), edited by Jill Burstein, Christy Doran, and Thamar Solorio, 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics. doi:10.18653/v1/N19-1423.
5. Doğan, Mehmet Ferit. 2025. “From Prompts to Precision: A Comparative Experimental Study of Translation Quality Outputs with Focus on Role-Based (Persona) Prompting for Medical Texts.” Master’s thesis, Istanbul 29 Mayıs University.
6. Gao, Yuan, Ruili Wang, and Feng Hou. 2023. “How to Design Translation Prompts for ChatGPT: An Empirical Study.” arXiv:2304.02182v2. doi:10.48550/arXiv.2304.02182.
7. International Organization for Standardization. 2015. ISO 17100:2015 — Translation Services — Requirements for Translation Services. Geneva: ISO. https://www.iso.org/standard/59149.html.
8. Jakobson, Roman. 1960. “Closing Statement: Linguistics and Poetics.” In Style in Language, edited by Thomas A. Sebeok, 350–377. New York: Wiley.

9. Lyu, Chenyang, Zefeng Du, Jitao Xu, Yitao Duan, Minghao Wu, Teresa Lynn, Alham Fikri Aji, et al. 2024. “A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models.” In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), edited by Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, 1339–1352. Torino, Italia: ELRA and ICCL. https://aclanthology.org/2024.lrec-main.120/.
10. Munday, Jeremy. 2016. Introducing Translation Studies: Theories and Applications. 4th ed. London: Routledge.
11. O’Brien, Sharon. 2024. “Human-Centered Augmented Translation: Against Antagonistic Dualisms.” In “Mean Machines? Sociotechnical (R)evolution and Human Labour in the Translation and Interpreting Industry,” edited by Michael Tieber and Stefan Baumgarten. Special Issue, Perspectives 32 (3): 391–406. doi:10.1080/0907676X.2023.2247423.
12. Öner, Işın, and Zehra Begüm Bengi. 2024. “Essential or Obsolete? The Role of Human Competencies in the Tech-Driven Language Services Industry.” transLogos Translation Studies Journal 7 (1): 78–104. doi:10.29228/transLogos.66.
13. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “BLEU: A Method for Automatic Evaluation of Machine Translation.” In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, 311–318. Philadelphia, Pennsylvania: Association for Computational Linguistics. doi:10.3115/1073083.1073135.
14. Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. “Language Models Are Unsupervised Multitask Learners.” https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
15. Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. “COMET: A Neural Framework for MT Evaluation.” In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2685–2702. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.213.
16. Sahoo, Pranab, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2025. “A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications.” arXiv:2402.07927v2. doi:10.48550/ARXIV.2402.07927.
17. Schumacher, Perrine. “Fostering MT Literacy and Reasserting the Value of Human Translators.” transLogos Translation Studies Journal 6 (2): 1–20. doi:10.29228/transLogos.57.
18. Scius-Bertrand, Anna, Michael Jungo, Lars Vögtlin, Jean-Marc Spat, and Andreas Fischer. 2024. “Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models.” arXiv:2412.13859v1. doi:10.48550/arXiv.2412.13859.
19. Sharkey, Edward, and Philip Treleaven. 2024. “Comparison of BERT vs GPT.” arXiv:2405.12990v1. doi:10.48550/ARXIV.2405.12990.
20. Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014. “Sequence to Sequence Learning with Neural Networks.” In Proceedings of the 27th International Conference on Neural Information Processing Systems, 3104–3112. https://papers.nips.cc/paper_files/paper/2014/hash/5a18e133cbf9f257297f410bb7eca942-Abstract.html.
21. Tieber, Michael, and Stefan Baumgarten. 2024. “Introduction: Mean Machines? Sociotechnical (R)evolution and Human Labour in the Translation and Interpreting Industry.” In “Mean Machines? Sociotechnical (R)evolution and Human Labour in the Translation and Interpreting Industry,” edited by Michael Tieber and Stefan Baumgarten. Special Issue, Perspectives 32 (3): 379–390. doi:10.1080/0907676X.2024.2346378.
22. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, et al. 2017. “Attention Is All You Need.” arXiv:1706.03762v1. doi:10.48550/arXiv.1706.03762.
23. Vlachopoulos, Stefanos. 2017. “Language for Specific Purposes and Translation.” In The Routledge Handbook of Translation Studies and Linguistics, edited by Kirsten Malmkjær, 425–440. London: Routledge.
24. Vöge, Lukas, Vincent Gurgul, and Stefan Lessmann. 2024. “Leveraging Zero-Shot Prompting for Efficient Language Model Distillation.” arXiv:2403.15886v1. doi:10.48550/arXiv.2403.15886.
25. Yamada, Masaru. 2024. “Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT’s Customizability.” arXiv:2308.01391v2. doi:10.48550/arXiv.2308.01391.
26. Zheng, Mingqian, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024. “When ‘A Helpful Assistant’ Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models.” arXiv:2311.10054v3. doi:10.48550/arXiv.2311.10054.