Osteoartrit hakkındaki popüler sorulara üç farklı yapay zekâ modelinin yanıtlarının ortopedi ve romatoloji uzmanları tarafından değerlendirilmesi

Abstract

Amaç: Bu çalışma, osteoartrit (OA) ile ilgili popüler hasta sorularına üç farklı yapay zekâ (YZ) modelinin (ChatGPT-3.5, DeepSeek-R1 ve Gemini-2.5) verdiği yanıtların doğruluk, tamlık ve okunabilirlik açısından karşılaştırılmasını amaçlamaktadır. Yöntem: Google Trends verilerinden elde edilen OA ile ilgili 12 hasta odaklı soru, üç YZ modeline yöneltildi. Yanıtlar, bir ortopedi ve bir romatoloji uzmanı tarafından doğruluk (1-6 puan), tamlık (1-3 puan) ve Flesch Okuma Kolaylığı puanı (FRES) açısından değerlendirildi. İstatistiksel analizlerde Friedman testi ve Cohen’in kappa katsayısı kullanıldı. Bulgular: Ortalama doğruluk puanları ChatGPT için 5,83 ± 0,39, DeepSeek için 5,67 ± 0,49, Gemini için 5,75 ± 0,45 olarak bulundu (p = 0,472). Ortalama tamlık puanları sırasıyla 2,75 ± 0,45, 2,58 ± 0,51 ve 2,75 ± 0,45 idi (p = 0,449). Değerlendiriciler arası uyum doğruluk için 0,90, tamlık için 0,88 olarak saptandı (her ikisi p<0,001). Okunabilirlik puanları ChatGPT için 47,2 (“çok zor”), DeepSeek için 27,9 (“çok zor”), Gemini için 55,8 (“zor”) idi. Nitel analizde, ChatGPT ve DeepSeek’in talep edilenden fazla ve ayrıntılı bilgi sunma eğiliminde olduğu, Gemini’nin ise daha kısa ve doğrudan yanıtlar verdiği belirlendi. Sonuç: YZ modelleri OA ile ilgili yüksek doğruluk ve tamlık düzeyinde bilgi sunabilmektedir. Ancak ChatGPT ve DeepSeek’in okunabilirlik düzeyi düşük sağlık okuryazarlığı olan bireyler için yetersiz olabilir. Bu nedenle, YZ çıktılarının hedef kitleye uygun dilde düzenlenerek, uzman denetimi ile sunulması hasta eğitimi ve bilgilendirme süreçlerinde kritik öneme sahiptir.

Keywords

• Osteoartrit
• yapay zeka
• ortopedi
• romatoloji

Supporting Institution

Yok.

Ethical Statement

Bu çalışmada insan veya hayvan denek kullanılmadığı için etik kurul onayı alınmamıştır.

Thanks

Yok.

Evaluation of responses to popular questions about osteoarthritis by three different artificial intelligence models assessed by orthopedic and rheumatology specialists

Abstract

Aim: This study aimed to compare the accuracy, completeness, and readability of responses generated by three different artificial intelligence (AI) models—ChatGPT-3.5, DeepSeek-R1, and Gemini-2.5—to popular patient questions about osteoarthritis (OA). Methods: Twelve patient-focused OA questions obtained from Google Trends were presented to each AI model. Responses were evaluated by an orthopedic surgeon and a rheumatologist for accuracy (1–6 points), completeness (1–3 points), and Flesch Reading Ease Score (FRES). Statistical analyses included the Friedman test and Cohen’s kappa coefficient. Results: Mean accuracy scores were 5.83 ± 0.39 for ChatGPT, 5.67 ± 0.49 for DeepSeek, and 5.75 ± 0.45 for Gemini (p = 0.472). Mean completeness scores were 2.75 ± 0.45, 2.58 ± 0.51, and 2.75 ± 0.45, respectively (p = 0.449). Inter-rater agreement was excellent, with kappa values of 0.90 for accuracy and 0.88 for completeness (both p<0.001). Readability scores were 47.2 (“very difficult”) for ChatGPT, 27.9 (“very difficult”) for DeepSeek, and 55.8 (“difficult”) for Gemini. Qualitative analysis revealed that ChatGPT and DeepSeek tended to provide more extensive and detailed information than requested, while Gemini provided shorter, more direct answers. Conclusion: AI models can deliver high levels of accuracy and completeness when answering OA-related patient questions. However, the lower readability of ChatGPT and DeepSeek outputs may limit accessibility for individuals with low health literacy. Adjusting AI-generated content to match the target audience’s literacy level and delivering it under expert supervision is essential for optimizing patient education and communication.

Keywords

• Osteoarthritis
• artificial intelligence
• orthopedics
• rheumatology

Supporting Institution

None.

Ethical Statement

Since no human or animal subjects were involved in this study, ethics committee approval was not required.

Thanks

None.

References

1. Hunter DJ, Bierma-Zeinstra S. Osteoarthritis. Lancet. 2019;393(10182):1745-1759.
2. Bijlsma JWJ, Berenbaum F, Lafeber FPJG. Osteoarthritis: an update with relevance for clinical practice. Lancet. 2011;377(9783):2115-2126.
3. Loeser RF, Collins JA, Diekman BO. Ageing and the pathogenesis of osteoarthritis. Nat Rev Rheumatol. 2016;12(7):412-420.
4. Biswas S. ChatGPT and the future of medical writing. Radiology. 2023;307(2):e223312.
5. Tang L, Sun Z, Idnay B, et al. Evaluating Large Language Models on Medical Evidence Summarization. medRxiv. Preprint posted online April 24, 2023. doi:10.1101/2023.04.22.23288967
6. Flesch R. A new readability yardstick. J Appl Psychol. 1948;32:221-233. doi:10.1037/h0057532
7. Çelik NÇ, Kılınç EA. Assessment of ChatGPT's adherence to EULAR diagnostic criteria and therapeutic protocols for rheumatoid arthritis at two distinct time points, 14 days apart, utilizing binary and multiple-choice inquiries. Clin Rheumatol. 2025;44(6):2233-2239.
8. Oruçoğlu N, Altunel Kılınç E. Performance of artificial intelligence chatbot as a source of patient information on anti-rheumatic drug use in pregnancy. J Surg Med. 2023;7(10):651-655.

9. Mohammadi S, Salehi MA, Jahanshahi A, et al. Artificial intelligence in osteoarthritis detection: a systematic review and meta-analysis. Osteoarthritis Cartilage. 2024;32(3):241-253.
10. Smolle MA, Goetz C, Maurer D, et al. Artificial intelligence–based computer-aided system for knee osteoarthritis assessment increases experienced orthopaedic surgeons' agreement rate and accuracy. Knee Surg Sports Traumatol Arthrosc. 2023;31(3):1053-1062.
11. Altunel Kılınç E, Çabuk Çelik N. Evaluation of artificial intelligence use in ankylosing spondylitis with ChatGPT-4: patient and physician perspectives. Clin Rheumatol. Published online September 11, 2025.
12. Cao M, Wang Q, Zhang X, et al. Large language models' performances regarding common patient questions about osteoarthritis: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Perplexity. J Sport Health Sci. 2024;14:101016.
13. Jayakumar P, Moore MG, Furlough KA, et al. Comparison of an artificial intelligence–enabled patient decision aid vs educational material on decision quality, shared decision-making, patient experience, and functional outcomes in adults with knee osteoarthritis: a randomized clinical trial. JAMA Netw Open. 2021;4(2):e2037107.
14. Yang Y, Lin J, Zhang J. The effects of ChatGPT on patient education of knee osteoarthritis: a preliminary study of 60 cases. Int J Surg. Published online May 22, 2025.
15. Yoo M, Jang CW. Presentation suitability and readability of ChatGPT's medical responses to patient questions about knee osteoarthritis. Health Inform J. 2025;31(1):14604582251315587.
16. Wu H, Yao S, Bao H, et al. ChatGPT-4.0 and DeepSeek-R1 does not yet provide clinically supported answers for knee osteoarthritis. Knee. 2025;56:386-396. doi:10.1016/j.knee.2025.06.007
17. Dagher T, Dwyer EP, Baker HP, Kalidoss S, Strelzow JA. “Dr. AI Will See You Now”: how do ChatGPT-4 treatment recommendations align with orthopaedic clinical practice guidelines? Clin Orthop Relat Res. 2024;482(12):2098-2106.
18. Turkish Statistical Institute (TUIK). Ulusal Eğitim İstatistikleri 2024. Accessed August 1, 2025. https://data.tuik.gov.tr/Bulten/Index?p=Ulusal-Egitim-Istatistikleri-2024-53937
19. Rowlands G, Protheroe J, Winkley J, Richardson M, Seed PT, Rudd R. A mismatch between population health literacy and the complexity of health information: an observational study. Br J Gen Pract. 2015;65(635):e379-e386.
20. Miller TA. Health literacy and adherence to medical treatment in chronic and acute illness: a meta-analysis. Patient Educ Couns. 2016;99(7):1079-1086.