EXPERT EVALUATION OF ARTIFICIAL INTELLIGENCE GENERATED ANSWERS TO FREQUENTLY ASKED QUESTIONS ABOUT RHINOPLASTY
Abstract
Aim: Large language models (LLMs) such as ChatGPT-4, DeepSeek, and Gemini are increasingly explored as tools for patient education and clinical decision support. However, concerns remain regarding their factual accuracy, completeness, and readability, especially when addressing frequently asked patient questions in postoperative care. This study aimed to directly compare three leading AI models—ChatGPT-4, DeepSeek, and Gemini—in terms of their accuracy, clarity, relevance, and completeness when answering common postoperative rhinoplasty FAQs. A secondary objective was to assess the readability of these AI-generated responses for a general patient audience.
Method: We selected 14 frequently asked questions based on authoritative AAO-HNS guidelines. Responses from each AI model were independently evaluated by 15 board-certified otorhinolaryngologists using a 5-point Likert scale across four domains: accuracy, clarity, relevance, and completeness. Readability was measured using the Flesch Reading Ease Score and Flesch–Kincaid Grade Level. Data were analyzed using appropriate statistical tests to identify significant differences among the models.
Results: Expert evaluations showed significant performance differences among the models. DeepSeek underperformed in both accuracy (p=0.00003) and completeness (p=0.0042) compared to ChatGPT-4 and Gemini. No statistically significant differences were observed for clarity (p=0.52) or relevance (p=0.42). Although readability scores did not significantly differ across models, all responses were deemed too complex for the average patient to fully understand.
Conclusion: While ChatGPT-4 and Gemini demonstrated higher accuracy and completeness than DeepSeek, none of the evaluated AI models produced content that met essential patient readability standards. These findings underscore the need for improved content accessibility and ongoing human oversight before LLMs can be reliably integrated into clinical patient education. This study establishes an important benchmark and highlights the urgency for future AI development to prioritize both factual integrity and true patient comprehension.
Keywords
Artificial Intelligence, Large Language Models, Rhinoplasty, Postoperative Care, Comprehension
References
- 1. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners [Preprint]. arXiv. 2020. arXiv:2005.14165.",
- 2. Vlad AL, Popazu C, Lescai AM, Voinescu DC, Balta AAŞ. The role of artificial intelligence in the diagnosis and management of rheumatoid arthritis. Medicina (Kaunas). 2025;61(4):689. doi:10.3390/medicina61040689.
- 3. Villarroel PP, Langdon C, Arancibia-Tagle D. Improving postsurgical management of rhinoplasty: a comprehensive review of existing literature. Facial Plast Surg. Epub 2025 Feb 14. doi:10.1055/a-2533-2651.
- 4. Serrano LP, Maita KC, Avila FR, Torres-Guzman RA, Garcia JP, Eldaly AS, et al. Benefits and challenges of remote patient monitoring as perceived by health care practitioners: a systematic review. Perm J. 2023;27(4):100–11. doi:10.7812/TPP/23.022.
- 5. Aliyeva A, Sari E, Alaskarov E, Nasirov R. Enhancing postoperative cochlear implant care with ChatGPT-4: a study on artificial intelligence (AI)-assisted patient education and support. Cureus. 2024;16(2):e53897. doi:10.7759/cureus.53897.
- 6. Ishii LE, Tollefson TT, Basura GJ, Rosenfeld RM, Abramson PJ, Chaiet SR, et al. Clinical practice guideline: improving nasal form and function after rhinoplasty. Otolaryngol Head Neck Surg. 2017;156(2 Suppl):S1–30. doi:10.1177/0194599816683153.",
- 7. de Vries PLM, Baud D, Baggio S, Ceulemans M, Favre G, Gerbier E, et al. Enhancing perinatal health patient information through ChatGPT – an accuracy study. PEC Innov. 2025;6:100381. doi:10.1016/j.pecinn.2025.100381.
- 8. Kirchner GJ, Kim RY, Weddle JB, Bible JE. Can artificial intelligence improve the readability of patient education materials? Clin Orthop Relat Res. 2023;481(11):2260–7. doi:10.1097/CORR.0000000000002668.
- 9. Huang Y, Shi R, Chen C, Zhou X, Zhou X, Hong J, et al. Evaluation of large language models for providing educational information in orthokeratology care. Cont Lens Anterior Eye. 2025;48(3):102384. doi:10.1016/j.clae.2025.102384.
- 10. Zhang B. Readability analysis of texts in college English textbooks and reading passages in CET-6. OALib. 2022;9:1–19. doi:10.4236/oalib.1109445.