Diagnosing retinal disorders with artificial intelligence: the role of large language models in interpreting pattern electroretinography data

Abstract

Aims: To evaluate the diagnostic accuracy of Claude-3, a large language model, in detecting pathological features and diagnosing retinitis pigmentosa and cone-rod dystrophy using pattern electroretinography data. Methods: A subset of pattern electroretinography measurements from healthy individuals, patients with retinitis pigmentosa and cone-rod dystrophy was randomly selected from the PERG-IOBA dataset. The pattern electroretinography and clinical data, including age, gender, visual acuities, were provided to Claude-3 for analysis and diagnostic predictions. The model’s accuracy was assessed in two scenarios: “first choice,” evaluating the accuracy of the primary differential diagnosis and “top 3,” evaluating whether the correct diagnosis was included within the top three differential diagnoses. Results: A total of 46 subjects were included in the study: 20 healthy individuals, 13 patients with retinitis pigmentosa, 13 patients with cone-rod dystrophy. Claude-3 achieved 100% accuracy in detecting the presence or absence of pathology. In the “first choice” scenario, the model demonstrated moderate accuracy in diagnosing retinitis pigmentosa (61.5%) and cone-rod dystrophy (53.8%). However, in the “top 3” scenario, the model’s performance significantly improved, with accuracies of 92.3% for retinitis pigmentosa and 76.9% for cone-rod dystrophy. Conclusion: This is the first study to demonstrate the potential of large language models, specifically Claude-3, in analyzing pattern electroretinography data to diagnose retinal disorders. Despite some limitations, the model’s high accuracy in detecting pathologies and distinguishing between specific diseases highlights the potential of large language models in ocular electrophysiology. Future research should focus on integrating multimodal data, and conducting comparative analyses with human experts.

Keywords

• Retinitis pigmentosa
• cone rod dystrophy
• pattern electroretinography
• large language models

Ethical Statement

Ethics Committee Approval: Since the PERG IOBA dataset from the PhysioNet database was used in this study, ethical approval is not required. The terms of use of the database have been adhered to. Informed Consent: Because the study was designed retrospectively, no written informed consent form was obtained from patients. Referee Evaluation Process: Externally peer-reviewed. Conflict of Interest Statement: The authors have no conflicts of interest to declare. Financial Disclosure: The authors declared that this study has received no financial support. Author Contributions: All of the authors declare that they have all participated in the design, execution, and analysis of the paper, and that they have approved the final version. Acknowledgement: Our research’s data was presented in ‘15th Medical Informatics Congress’ as ‘Oral Presentation’ on May 30, 2024.

References

1. Thompson DA, Bach M, McAnany JJ, Šuštar Habjan M, Viswanathan S, Robson AG. ISCEV standard for clinical pattern electroretinography (2024 update). Doc Ophthalmol. 2024; 148(2):75-85. doi:10.1007/s10633-024-09970-1
2. Robson AG, El-Amir A, Bailey C, et al. Pattern ERG correlates of abnormal fundus autofluorescence in patients with retinitis pigmentosa and normal visual acuity. Invest Ophthalmol Vis Sci. 2003;44(8):3544-3550. doi:10.1167/iovs.02-1278
3. Gallo Afflitto G, Chou TH, Swaminathan SS, et al. Pattern electroretinogram in ocular hypertension, glaucoma suspect and early manifest glaucoma eyes: a systematic review and meta-analysis. Ophthalmol Sci. 2023;3(4):100322. doi:10.1016/j.xops. 2023.100322
4. Janáky M, Pálffy A, Horváth G, Tuboly G, Benedek G. Pattern-reversal electroretinograms and visual evoked potentials in retinitis pigmentosa. Doc Ophthalmol. 2008;117(1):27-36. doi:10. 1007/s10633-007-9099-0
5. Robson AG, Nilsson J, Li S, et al. ISCEV guide to visual electrodiagnostic procedures. Doc Ophthalmol. 2018;136(1):1-26. doi:10.1007/s10633-017-9621-y
6. Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018;2(10):719-731. doi:10.1038/s41551-018-0305-z
7. Ting DSW, Pasquale LR, Peng L, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. 2019;103(2): 167-175. doi:10.1136/bjophthalmol-2018-313173
8. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama. 2016;316(22):2402-2410. doi:10.1001/jama.2016.17216

9. Asaoka R, Murata H, Hirasawa K, et al. Using deep learning and transfer learning to accurately diagnose early-onset glaucoma from macular optical coherence tomography images. Am J Ophthalmol. 2019;198:136-145. doi:10.1016/j.ajo.2018.10.007
10. Liu X, Faes L, Kale AU, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health. 2019;1(6):e271-e297. doi:10.1016/S2589-7500(19)30123-2
11. McKinney SM, Sieniek M, Godbole V, et al. International evaluation of an AI system for breast cancer screening. Nature. 2020;577(7788):89-94. doi:10.1038/s41586-019-1799-6
12. Char DS, Abràmoff MD, Feudtner C. Identifying ethical considerations for machine learning healthcare applications. Am J Bioeth. 2020;20(11):7-17. doi:10.1080/15265161.2020.1819469
13. Raffel C, Shazeer NM, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res. 2019;21(140):1-67.
14. Head CB, Jasper P, McConnachie M, Raftree L, Higdon G. Large language model applications for evaluation: opportunities and ethical implications. N Direct Evaluat. 2023;2023(178-179):33-46. doi:10.1002/ev.20556
15. Meng X, Yan X, Zhang K, et al. The application of large language models in medicine: a scoping review. iScience. 2024;27(5): 109713. doi:10.1016/j.isci.2024.109713
16. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. doi:10.1038/s41586-023-06291-2
17. Wu J, Ma Y, Wang J, Xiao M. The application of chatgpt in medicine: a scoping review and bibliometric analysis. J Multidiscip Healthc. 2024;17:1681-1692. doi:10.2147/JMDH.S463128
18. Yap GH, Chen LY, Png R, et al. Clinical value of electrophysiology in determining the diagnosis of visual dysfunction in neuro-ophthalmology patients. Doc Ophthalmol. 2015;131(3):189-96. doi:10.1007/s10633-015-9515-9
19. Goldberger AL, Amaral LA, Glass L, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation. 2000; 101(23):E215-220. doi:10.1161/01.cir.101.23.e215
20. Fernández I, Cuadrado Asensio R, Larriba Y, Rueda C, Coco-Martin RM. A comprehensive dataset of pattern electroretinograms for ocular electrophysiology research: the PERG-IOBA dataset (version 1.0.0). PhysioNet. 2024. doi:10. 13026/d24m-w054
21. Bach M, Brigell MG, Hawlina M, et al. ISCEV standard for clinical pattern electroretinography (PERG): 2012 update. Doc Ophthalmol. 2013;126(1):1-7. doi:10.1007/s10633-012-9353-y
22. Parikh R, Mathai A, Parikh S, Chandra Sekhar G, Thomas R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol. 2008;56(1):45-50. doi:10.4103/0301-4738.37595
23. Popović P, Jarc-Vidmar M, Hawlina M. Abnormal fundus autofluorescence in relation to retinal function in patients with retinitis pigmentosa. Graefes Arch Clin Exp Ophthalmol. 2005; 243(10):1018-1027. doi:10.1007/s00417-005-1186-x
24. Hamel CP. Cone rod dystrophies. Orphanet J Rare Dis. 2007;2:7. doi:10.1186/1750-1172-2-7
25. Downes SM, Payne AM, Kelsell RE, et al. Autosomal dominant cone-rod dystrophy with mutations in the guanylate cyclase 2D gene encoding retinal guanylate cyclase-1. Arch Ophthalmol (Chicago, Ill : 1960). 2001;119(11):1667-1673. doi:10.1001/archopht.119.11.1667
26. Schwartz IS, Link KE, Daneshjou R, Cortés-Penfield N. Black box warning: large language models and the future of infectious diseases consultation. Clin Infect Dis. 2024;78(4):860-866. doi:10. 1093/cid/ciad633
27. Harrer S. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine. EBioMedicine. 2023;90:104512. doi:10.1016/j.ebiom.2023.104512
28. Au Yeung J, Kraljevic Z, Luintel A, et al. AI chatbots not yet ready for clinical use. Frontiers in digital health. 2023;5:1161098. doi:10.3389/fdgth.2023.1161098
29. Rojas-Carabali W, Sen A, Agarwal A, et al. Chatbots Vs. Human experts: evaluating diagnostic performance of chatbots in uveitis and the perspectives on ai adoption in ophthalmology. Ocul Immunol Inflamm. 2023:1-8. doi:10.1080/09273948.2023.2266730