Büyük Dil Modellerinin Yan Dal Anesteziyoloji Sınav Sorularındaki Doğruluğu

Öz

Amaç: Büyük dil modeli (BDM) sınav çalışmalarında aynı maddeler modellerce tekrar tekrar yanıtlandığından gözlemler kümelenmiştir ve alışılmış aralıklar kesinliği olduğundan yüksek gösterir. Yedi BDM'nin yan dal düzeyinde bir anesteziyoloji soru bankasındaki doğruluğunu ölçmeyi ve modeller arası farklar ile alt grup farklarından hangilerinin kümelenmeyi gözeten bir analizde ayırt edilebilir kaldığını belirlemeyi amaçladık. Gereç ve Yöntem: Anesteziyoloji ve Reanimasyon Yan Dal Sınavı tarzında, yazarlar tarafından yazılan ve güncel kılavuzlar ile ders kitapları üzerinden denetlenen 120 maddelik beş seçenekli bir soru bankası, yedi BDM'ye sıcaklık 0'da ve her denemede seçenek sırası yeniden rastgeleleştirilerek beş denemede soruldu (120 benzersiz maddeden 4.200 yanıt). Doğruluk, maddeleri yeniden örnekleyen küme önyüklemesi %95 güven aralıkları (GA; 4.000 yineleme) ve karşılaştırma için naif Wilson aralıklarıyla sunuldu; alt grup analizleri keşifseldir. Bulgular: Birleşik doğruluk %80,1 idi (küme-dayanıklı %95 GA 75,8-83,9; naif Wilson 78,8-81,3). Modeller arası farklar büyük ve dayanıklıydı: %60,8 (54,0-67,5) ile %94,0 (90,3-97,0); uçların aralıkları örtüşmedi. Küme-dayanıklı aralıklar 26 tahminin 25'inde daha genişti (ortanca 2,1 kat; aralık 0,8-4,0). Alt grup karşılaştırmalarının çoğu ayırt edilebilir değildi, aralıklar belirgin örtüştü: vinyetli (%75,4; 62,5-86,5) ile vinyetsiz (%80,9; 76,6-84,9) ve kılavuz bağımlı (%73,0; 64,6-80,7) ile diğer maddeler (%80,9; 76,6-84,9). Yalnızca en zayıf ve en güçlü alanlar arasındaki karşıtlıklar korundu. Denemeler arası standart sapma 1,26-2,80 puandı. Sonuç: Modeller arası farklar ve denemeler arası kararsızlık dayanıklıdır; sıkça vurgulanan alan ve madde niteliği farkları, kümelenme gözetildiğinde çoğunlukla ayırt edilebilir değildir. 120 maddelik ölçütler doğrulukları belirgin ayrışan modelleri ayırt edebilir ancak zayıflıklarını güvenilir biçimde yerelleştiremez.

Anahtar Kelimeler

• Yapay zekâ
• Anesteziyoloji
• Eğitimsel ölçme
• Klinik karar verme
• Doğal dil işleme

Etik Beyan

Bu çalışmada insan katılımcı, hayvan, hasta verisi ya da kimlik belirlemeye elverişli herhangi bir kişisel bilgi kullanılmamıştır. Araştırma yalnızca Ölçme, Seçme ve Yerleştirme Merkezi (ÖSYM) tarafından kamuya açık biçimde yayımlanan çoktan seçmeli sınav soruları ile cevap anahtarları ve herkesin erişimine açık hesaplamalı dil modellerinin ürettiği çıktılar üzerinde yürütülmüştür. İnsan veya hayvan deneği ile klinik veri söz konusu olmadığından kurumsal etik kuruldan onay alınması gerekmemiş, bilgilendirilmiş onam da uygulanabilir olmamıştır. Çalışma, insan deneği içermeyen araştırmalar bakımından geçerli olduğu ölçüde Helsinki Bildirgesi ilkelerine uygun olarak gerçekleştirilmiştir.

Large Language Model Accuracy on Subspecialty Anaesthesiology Examination Questions

Öz

Aim: In large language model (LLM) examination studies, models answer the same items repeatedly, so observations are clustered and conventional intervals overstate precision. We aimed to measure the accuracy of seven LLMs on a subspecialty-level anaesthesiology question bank and to determine which between-model and subgroup differences remain distinguishable under a cluster-respecting analysis. Materials and Methods: A 120-item, five-option multiple-choice bank in the style of the Anaesthesiology and Reanimation Subspecialty Examination, written by the authors and reviewed against current guidelines and textbooks, was administered to seven LLMs in five runs at temperature 0, with option order re-randomised per run (4,200 responses from 120 unique items). Accuracy is reported with cluster bootstrap 95% confidence intervals (CIs) resampling items (4,000 replicates) and naive Wilson intervals for comparison; subgroup analyses are exploratory. Results: Pooled accuracy was 80.1% (cluster-robust 95% CI 75.8-83.9; naive Wilson 78.8-81.3). Between-model differences were large and robust, spanning 60.8% (54.0-67.5) to 94.0% (90.3-97.0) with non-overlapping extremes. Cluster-robust intervals were wider in 25 of 26 estimates (median factor 2.1, range 0.8-4.0). Most subgroup comparisons were not distinguishable, intervals overlapping substantially: vignette (75.4%, 62.5-86.5) versus non-vignette (80.9%, 76.6-84.9) and guideline-dependent (73.0%, 64.6-80.7) versus other items (80.9%, 76.6-84.9). Only contrasts between weakest and strongest domains persisted. Between-run standard deviation was 1.26-2.80 points. Conclusion: Between-model differences and run-to-run instability are robust; commonly emphasised domain and item-characteristic differences are mostly not distinguishable once clustering is respected. Benchmarks of 120 items can separate models whose accuracies differ widely but cannot reliably localise their weaknesses.

Anahtar Kelimeler

• Artificial Intelligence
• Anesthesiology
• Educational Measurement
• Clinical Decision-Making
• Natural Language Processing

Etik Beyan

This study did not involve human participants, animals, patient data or any identifiable personal information. It was conducted exclusively on publicly released multiple-choice examination items and answer keys published by the Measuring, Selection and Placement Centre (ÖSYM), together with the outputs generated by publicly accessible computational language models. Because no human or animal subjects and no clinical data were involved, approval by an institutional ethics committee was not required and informed consent was not applicable. The study was conducted in accordance with the principles of the Declaration of Helsinki insofar as they apply to research not involving human subjects.

Kaynakça

1. 1. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med 2023;29(8):1930-40.
2. 2. Kung TH, Cheatham M, Medenilla A et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2(2):e0000198.
3. 3. Gilson A, Safranek CW, Huang T et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? JMIR Med Educ 2023;9:e45312.
4. 4. Singhal K, Azizi S, Tu T et al. Large language models encode clinical knowledge. Nature 2023;620(7972):172-80.
5. 5. Vishwanath K, Alyakin A, Ghosh M et al. Evaluating the performance and fragility of large language models on the self-assessment for neurological surgeons. Neurosurgery 2025. doi:10.1227/neu.0000000000003878.
6. 6. Angel MC, Rinehart JB, Cannesson MP, Baldi P. Clinical knowledge and reasoning abilities of AI large language models in anesthesiology: a comparative study on the American Board of Anesthesiology examination. Anesth Analg 2024;139(2):349-56.
7. 7. Altermatt FR, Neyem A, Sumonte NI, Villagrán I, Mendoza M, Lacassie HJ. Evaluating the performance of large language models on the CONACEM anesthesiology certification exam: a comparison with human participants. Appl Sci 2025;15(11):6245.
8. 8. Ronen A, Fein S, Orbach-Zinger S et al. Large language models versus human examinee performance on Israeli anesthesiology board examinations. Sci Rep 2026;16(1):14978.

9. 9. Hashimoto DA, Witkowski E, Gao L, Meireles O, Rosman G. Artificial intelligence in anesthesiology: current techniques, clinical applications, and limitations. Anesthesiology 2020;132(2):379-94.
10. 10. Epstein RH, Dexter F. Variability in large language models' responses to medical licensing and certification examinations. JMIR Med Educ 2023;9:e48305.
11. 11. Zheng C, Zhou H, Meng F, Zhou J, Huang M. Large language models are not robust multiple choice selectors. In: Proceedings of the Twelfth International Conference on Learning Representations (ICLR), Vienna, 2024.
12. 12. Pezeshkpour P, Hruschka E. Large language models sensitivity to the order of options in multiple-choice questions. In: Findings of the Association for Computational Linguistics: NAACL 2024. Mexico City: Association for Computational Linguistics, 2024:2006-17.
13. 13. Xu C, Guan S, Greene D, Kechadi MT. Benchmark data contamination of large language models: a survey. arXiv 2024;arXiv:2406.04244.
14. 14. Acar A. Data contamination in AI evaluation. JMIR Med Inform 2025;13:e80987.
15. 15. Republic of Türkiye Ministry of Health, Medical Specialty Board (TUK). Anaesthesiology and Reanimation specialty training core curriculum, v.2.5. Ankara: TUK [cited 29 Jul 2026]. Available from: https://tuk.saglik.gov.tr/TR-87040/v-25-dokumanlari.html
16. 16. Measuring, Selection and Placement Centre (ÖSYM). 2026 Subspecialty Training Entrance Examination (YDUS) application guide. Ankara: ÖSYM; 2026. Available at: https://www.osym.gov.tr (accessed 29 July 2026).
17. 17. The American Board of Anesthesiology. Initial certification in anesthesiology content outline. Raleigh: ABA; 2023 [cited 29 Jul 2026]. Available from: https://www.theaba.org
18. 18. European Society of Anaesthesiology and Intensive Care. EDAIC syllabus. Brussels: ESAIC [cited 29 Jul 2026]. Available from: https://www.esaic.org
19. 19. National Board of Medical Examiners. NBME item-writing guide: constructing written test questions for the health sciences. 6th ed. Philadelphia: NBME; 2020.
20. 20. Haladyna TM, Downing SM, Rodriguez MC. A review of multiple-choice item-writing guidelines for classroom assessment. Appl Meas Educ 2002;15(3):309-34.
21. 21. Apfelbaum JL, Hagberg CA, Connis RT et al. 2022 American Society of Anesthesiologists practice guidelines for management of the difficult airway. Anesthesiology 2022;136(1):31-81.
22. 22. Joshi GP, Abdelmalak BB, Weigel WA et al. 2023 American Society of Anesthesiologists practice guidelines for preoperative fasting: carbohydrate-containing clear liquids with or without protein, chewing gum, and pediatric fasting duration. Anesthesiology 2023;138(2):132-51.
23. 23. Wigginton JG, Agarwal S, Bartos JA et al. Part 9: adult advanced life support: 2025 American Heart Association guidelines for cardiopulmonary resuscitation and emergency cardiovascular care. Circulation 2025;152(Suppl 2):S538-77.
24. 24. Soar J, Böttiger BW, Carli P et al. European Resuscitation Council guidelines 2025: adult advanced life support. Resuscitation 2025;215(Suppl 1):110769.
25. 25. Kopp SL, Vandermeulen E, McBane RD, Perlas A, Leffert L, Horlocker T. Regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy: American Society of Regional Anesthesia and Pain Medicine evidence-based guidelines (fifth edition). Reg Anesth Pain Med 2025. doi:10.1136/rapm-2024-105766.
26. 26. Gropper MA, Eriksson LI, Fleisher LA, Cohen NH, Leslie K, Johnson-Akeju O, editors. Miller's anesthesia. 10th ed. Philadelphia: Elsevier; 2025.
27. 27. Butterworth JF, Mackey DC, Wasnick JD. Morgan & Mikhail's clinical anesthesiology. 7th ed. New York: McGraw Hill; 2022.
28. 28. Cullen BF, Stock MC, Ortega R et al, editors. Barash, Cullen, and Stoelting's clinical anesthesia. 9th ed. Philadelphia: Wolters Kluwer; 2023.
29. 29. Flood P, Rathmell JP, Urman RD, editors. Stoelting's pharmacology and physiology in anesthetic practice. 6th ed. Philadelphia: Wolters Kluwer; 2022.
30. 30. Anderson LW, Krathwohl DR, editors. A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives. New York: Longman; 2001.
31. 31. Efron B, Tibshirani RJ. An introduction to the bootstrap. New York: Chapman and Hall; 1993.
32. 32. Cameron AC, Gelbach JB, Miller DL. Bootstrap-based improvements for inference with clustered errors. Rev Econ Stat 2008;90(3):414-27.
33. 33. Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22(158):209-12.
34. 34. Brown LD, Cai TT, DasGupta A. Interval estimation for a binomial proportion. Stat Sci 2001;16(2):101-33.
35. 35. Tsui BCH, Kropelin B, Ganapathy S, Finucane B. Dextrose 5% in water: fluid medium maintaining electrical stimulation of peripheral nerve during stimulating catheter placement. Acta Anaesthesiol Scand 2005;49(10):1562-5.
36. 36. Tsui BCH, Kropelin B. The electrophysiological effect of dextrose 5% in water on single-shot peripheral nerve stimulation. Anesth Analg 2005;100(6):1837-9.
37. 37. Eger EI 2nd. New inhaled anesthetics. Anesthesiology 1994;80(4):906-22.