Clinical Decision Support via ChatGPT: An Evidence-Based Perspective in Pediatric Occupational Therapy

Abstract

Aim: This study aims to evaluate the accuracy of ChatGPT’s responses to clinical questions related to Cerebral Palsy (CP) and Autism Spectrum Disorder (ASD) in the field of occupational therapy, as well as the reliability of the references it provides.

Method: Ten clinical questions were formulated, and answers were prepared based on clinical guidelines and reviewed by two expert clinicians. The same questions were presented to ChatGPT, and its responses and references were rated by two independent evaluators using a four-point Likert scale. Weighted Kappa Coefficients were calculated to assess inter-rater agreement.

Results: ChatGPT’s responses demonstrated high accuracy, with strong inter-rater agreement (Weighted Kappa: 0.72–0.75). However, significant deficiencies were identified in reference reliability, with moderate to high inter-rater agreement (Weighted Kappa: 0.75–0.78). Additionally, fictitious reference rates ranged from 40% to 100% for CP-related questions and from 25% to 100% for ASD-related questions.

Conclusion: While ChatGPT shows potential as a clinical decision support tool in occupational therapy, the limitations in reference accuracy restrict its reliability in healthcare applications. Future studies should focus on improving artificial intelligence models’ reference accuracy to enhance their applicability in healthcare settings.

Keywords

• Artificial intelligence
• autism spectrum disorder
• cerebral palsy
• occupational therapy

Supporting Institution

This research received no external funding or institutional support.

Ethical Statement

This study did not require ethics committee approval since it did not involve human participants, animal experiments, or any data obtained directly from individuals.

ChatGPT ile Klinik Karar Desteği: Pediatrik Ergoterapide Kanıta Dayalı Bir Perspektif

Abstract

Amaç: Bu çalışmanın amacı, Ergoterapi alanında Serebral Palsi (SP) ve Otizm Spektrum Bozukluğu (OSB) ile ilgili klinik sorulara ChatGPT’nin verdiği yanıtların doğruluğunu ve sağladığı kaynakların güvenilirliğini değerlendirmektir.

Yöntem: On klinik soru oluşturulmuş, yanıtlar klinik rehberlere dayalı olarak hazırlanmış ve iki uzman klinisyen tarafından gözden geçirilmiştir. Aynı sorular ChatGPT’ye yöneltilmiş, yanıtları ve kaynakları iki bağımsız değerlendirici tarafından dört dereceli Likert ölçeği ile puanlanmıştır. Değerlendiriciler arası uyumu belirlemek için Ağırlıklı Kappa Katsayıları hesaplanmıştır.

Bulgular: ChatGPT’nin yanıtları yüksek doğruluk göstermiş ve değerlendiriciler arası güçlü uyum saptanmıştır (Ağırlıklı Kappa: 0,72–0,75). Ancak, kaynak güvenilirliğinde önemli eksiklikler bulunmuş olup, değerlendiriciler arası uyum orta-yüksek düzeydeydi (Ağırlıklı Kappa: 0,75–0,78). Ayrıca, hayali kaynak oranları SP ile ilgili sorularda %40–100, OSB ile ilgili sorularda %25–100 arasında değişmiştir.

Sonuç: ChatGPT, ergoterapi alanında klinik karar destek aracı olarak potansiyel göstermektedir; ancak kaynak doğruluğundaki sınırlılıklar, sağlık hizmetlerindeki güvenilirliğini kısıtlamaktadır. Gelecek çalışmalarda, yapay zekâ modellerinin kaynak doğruluğunun geliştirilmesine odaklanılması, sağlık alanındaki uygulanabilirliklerini artıracaktır.

Keywords

• Yapay zekâ
• otizm spektrum bozukluğu
• serebral palsi
• ergoterapi

Supporting Institution

Bu araştırma herhangi bir kurum veya kuruluş tarafından desteklenmemiştir.

Ethical Statement

Bu çalışma, insan katılımcılar, hayvan deneyleri veya doğrudan bireylerden elde edilen herhangi bir veri içermediğinden etik kurul onayı gerektirmemektedir.

References

1. 1. Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on medical challenge problems. arXiv. 2023. doi: 10.48550/arXiv.2303.13375.
2. 2. Liu Y, Han T, Ma S, et al. Summary of ChatGPT-related research and perspective towards the future of large language models. Meta-Radiology. 2023;100:017.
3. 3. Hu K. ChatGPT sets record for fastest-growing user base - analyst note. Reuters. February 2, 2023. Accessed February 1, 2023. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01.
4. 4. Campbell WA IV, Chick JF, Shin D, Makary MS. Understanding ChatGPT for evidence-based utilization in interventional radiology. Clin Imaging. 2024;110:098.
5. 5. Rao A, Pang M, Kim J. Assessing the utility of ChatGPT throughout the entire clinical workflow. medRxiv. 2023.
6. 6. Garg RK, Urs VL, Agarwal AA, Chaudhary SK, Paliwal V, Kar SK. Exploring the role of ChatGPT in patient care (diagnosis and treatment) and medical research: A systematic review. Health Promot Perspect. 2023;13(3):183.
7. 7. Von Ende E, Ryan S, Crain MA, Makary MS. Artificial intelligence, augmented reality, and virtual reality advances and applications in interventional radiology. Diagnostics. 2023;13(5):892.
8. 8. Greengard S. ChatGPT: Understanding the ChatGPT AI chatbot. Eweek. December 9, 2022. https://www.eweek.com/big-data-and-analytics/chatgpt/.

9. 9. Manning CD. Human language understanding & reasoning. Proc Am Acad Arts Sci. 2022;151(2):127-138. doi: 10.1162/daed_a_01905.
10. 10. Mündler N, He J, Jenko S, Vechev M. Self-contradictory hallucinations of large language models: Evaluation, detection, and mitigation. arXiv. 2023.
11. 11. Hoffmann T, Bennett S, Del Mar C. Evidence-based practice across the health professions. Philadelphia, PA: Elsevier Health Sciences; 2023.
12. 12. Sharma N. Use of AI in pediatric occupational therapy: A review. Rehabil Sci. 2024;9(2):21-26. doi: 10.11648/j.rs.20240902.12
13. 13. Berrezueta-Guzman S, Kandil M, Martín-Ruiz ML, de la Cruz IP, Krusche S. Exploring the efficacy of robotic assistants with ChatGPT and Claude in enhancing ADHD therapy: Innovating treatment paradigms. In: 2024 International Conference on Intelligent Environments (IE). 2024:25-32. doi: 10.1109/IE61493.2024.10599903.
14. 14. Morgan C, Fetters L, Adde L, et al. Early intervention for children aged 0 to 2 years with or at high risk of cerebral palsy: International clinical practice guideline based on systematic reviews. JAMA Pediatr. 2021;175(8):846-858. doi: 10.1001/jamapediatrics.2021.0878.
15. 15. Steinbrenner JR, Hume K, Odom SL, et al. Evidence-based practices for children, youth, and young adults with autism. Chapel Hill, NC: The University of North Carolina at Chapel Hill, Frank Porter Graham Child Development Institute, National Clearinghouse on Autism Evidence and Practice Review Team; 2020.
16. 16. Cahill SM, Beisbier S. Occupational therapy practice guidelines for children and youth ages 5–21 years. Am J Occup Ther. 2020;74(4):7404397010p1-7404397010p48.
17. 17. Kundel HL, Polansky M. Measurement of observer agreement. Radiology. 2003;228:303-308.
18. 18. Scaff SP, Reis FJ, Ferreira GE, Jacob MF, Saragiotto BT. Assessing the performance of AI chatbots in answering patients' common questions about low back pain. Ann Rheum Dis. 2025;84(1):143-149.
19. 19. Sawamura T, Lee H, Martinez R. Assessing the reliability of ChatGPT-generated references in physiotherapy clinical decision-making. Physiother Res J. 2024;32(2):89-101.
20. 20. Kaplan M, Çakar F, Bingöl H. Sağlık alanında yapay zekânın kullanımı: Derleme. Muş Alparslan Univ Sağlık Bilim Derg. 2024;4(3):75-85.
21. 21. Akalin B, Veranyurt Ü. Sağlık bilimlerinde yapay zekâ tabanlı klinik karar destek sistemleri. Gevher Nesibe J Med Health Sci. 2022;7(18):64-73.
22. 22. Graf A, Smith B, Johnson C. Evaluating the accuracy of AI-generated references in obstetric randomized controlled trials: A comparative study of ChatGPT, Google Bard, and Chatsonic. J Obstet Res. 2024;45(3):214-226.
23. 23. Kim S. In the era of ChatGPT, can medical artificial intelligence replace the doctor? Korean J Med. 2023;98(3).
24. 24. Waisberg E, Ong J, Masalkhi M, et al. GPT-4: A new era of artificial intelligence in medicine. Ir J Med Sci. 2023.
25. 25. Kunt MM. Tıpta dil tabanlı yapay zeka kullanımı. Anatolian J Emerg Med. 2023;6(3):137-140.