SCENARIO-BASED PREHOSPITAL TRIAGE OF CARBON MONOXIDE POISONING: COMPARING FIVE LARGE LANGUAGE MODELS WITH EMERGENCY MEDICAL SERVICES PERSONNEL IN A PROSPECTIVE STUDY

Abstract

Objective: Carbon monoxide (CO) poisoning requires rapid identification and timely decisions regarding the need for hyperbaric oxygen therapy (HBOT) to improve clinical outcomes. This study aimed to compare the decision-making performance of emergency medical services (EMS) personnel and large language models (LLMs) in accurately determining the need for HBOT during the prehospital phase of CO poisoning cases, and to explore the potential implementation of LLMs as decision-support tools for patient triage and referral. Material and Methods: In this prospective, simulation-based diagnostic accuracy study, 128 standardized scenario-based clinical cases (64 requiring HBOT, 64 not requiring HBOT) were developed based on established indications. Sixty EMS personnel and five LLMs (GPT-4o, GPT-4.5, GPT-o3, Gemini 2.5 Pro, and DeepSeek-R1) evaluated each scenario. Their responses were compared with the gold standard answers established by consensus between a medical toxicologist and a hyperbaric medicine specialist. Results: The overall accuracy of the EMS personnel was 65.1%, whereas the highest accuracy was observed with GPT-o3 (96.9%). All LLMs demonstrated 100% sensitivity, although the specificity varied, ranging from 26.6% (GPT-4.5) to 93.8% (GPT-o3). The accuracy of GPT-o3 (96.9%, p<0.001), GPT-4o (78.9%, p=0.001), and Gemini 2.5 Pro (75.8%, p=0.012) was significantly higher than that of the EMS personnel. GPT-o3 also significantly outperformed all other models (all p<0.001). Conclusion: Our study suggests that LLMs can support prehospital decision-making, facilitating timely referral to appropriate centers, improving early access to HBOT, and potentially enhancing outcomes. However, these findings are based on standardized simulation scenarios, and real-world clinical validation is necessary before widespread implementation.

Keywords

• Artificial intelligence
• carbon monoxide
• emergency medical service
• poisoning
• triage.

Supporting Institution

None.

Ethical Statement

The study was approved by the Scientific Research Ethics Committee of Karadeniz Technical University (Date: 07/04/2025, Number: 2025/96). The study was conducted in accordance with the Declaration of Helsinki. Participants received no payment or other compensation for their participation in the survey. Written informed consent was obtained from all participants.

Thanks

We would like to express our sincere gratitude to Dr. Evin Koç, a physician in Underwater and Hyperbaric Medicine, for her valuable contributions in establishing the gold standard decisions regarding the need for hyperbaric oxygen therapy in this study.

Karbonmonoksit Zehirlenmesinde Senaryo Tabanlı Hastane Öncesi Triyaj: Beş Büyük Dil Modeli ile Acil Sağlık Hizmetleri Personelinin Prospektif Karşılaştırılması

Abstract

Amaç: Karbonmonoksit (CO) zehirlenmesi, klinik sonuçları iyileştirmek için hızlı tanımlama ve hiperbarik oksijen tedavisi (HBOT) ihtiyacına yönelik zamanında karar verilmesini gerektirir. Bu çalışma, CO zehirlenmesi vakalarında hastane öncesi evrede HBOT ihtiyacının doğru şekilde belirlenmesinde acil sağlık hizmetleri (ASH) personeli ile büyük dil modellerinin (LLM) karar verme performansını karşılaştırmayı ve LLM'lerin hasta triyajı ve sevkinde karar destek aracı olarak kullanım potansiyelini kullanımını araştırmayı amaçlamıştır. Gereç ve Yöntemler: Bu prospektif ve simülasyon temelli tanısal doğruluk çalışmasında, yerleşik endikasyonlara dayalı olarak 128 standart vaka temelli klinik senaryo (64'ü HBOT gerektiren, 64'ü gerektirmeyen) geliştirildi. Altmış ASH personeli ve beş LLM (GPT-4o, GPT-4.5, GPT-o3, Gemini 2.5 Pro ve DeepSeek-R1) her bir senaryoyu değerlendirdi. Katılımcıların yanıtları, bir tıbbi toksikolog ve bir hiperbarik tıp uzmanı arasındaki fikir birliği ile oluşturulan altın standart cevaplarla karşılaştırıldı. Bulgular: ASH personelinin genel doğruluk oranı %65,1 iken, en yüksek doğruluk oranı GPT-o3 modelinde (%96,9) gözlemlenmiştir. Tüm LLM'ler %100 duyarlılık göstermekle birlikte özgüllük oranları %26,6 (GPT-4.5) ile %93,8 (GPT-o3) arasında değişmiştir. GPT-o3 (%96,9, p<0,001), GPT-4o (%78,9, p=0,001) ve Gemini 2.5 Pro (%75,8, p=0,012) modellerinin doğruluk oranları ASH personeline kıyasla istatistiksel olarak anlamlı düzeyde yüksek bulunmuştur. Ayrıca GPT-o3, diğer tüm modellerden anlamlı derecede daha iyi performans göstermiştir (tümü için p<0,001). Sonuç: Çalışmamız, LLM'lerin hastane öncesi karar verme süreçlerini destekleyebileceğini, uygun merkezlere zamanında sevki kolaylaştırabileceğini, HBOT'ye erken erişimi artırabileceğini ve potansiyel olarak klinik sonuçları iyileştirebileceğini öne sürmektedir. Bununla birlikte, bu bulgular standart simülasyon senaryolarına dayanmaktadır ve yaygın uygulama öncesinde gerçek dünya klinik doğrulamasının yapılması gerekmektedir.

Keywords

• Yapay zeka
• karbonmonoksit
• acil sağlık hizmeti
• zehirlenme
• triyaj.

References

1. Mattiuzzi C, Lippi G. Worldwide epidemiology of carbon monoxide poisoning. Hum Exp Toxicol. 2020;39(4):387-392.
2. Liao SC, Shao SC, Yang KJ, Yang CC. Real-world effectiveness of hyperbaric oxygen therapy for delayed neuropsychiatric sequelae after carbon monoxide poisoning. Sci Rep. 2021;11(1):19212.
3. Arslan A. Hyperbaric oxygen therapy in carbon monoxide poisoning in pregnancy: Maternal and fetal outcome. Am J Emerg Med. 2021;43:41-45.
4. Weaver LK, Hopkins RO, Chan KJ, et al. Hyperbaric oxygen for acute carbon monoxide poisoning. N Engl J Med. 2002;347(14):1057-1067.
5. Lee Y, Cha YS, Kim SH, Kim H. Effect of hyperbaric oxygen therapy initiation time in acute carbon monoxide poisoning. Crit Care Med. 2021;49(10):e910-e919.
6. Jia Y, Han N, Guo H, et al. Hyperbaric oxygen therapy for acute carbon monoxide poisoning patients with coma onset. Eur J Med Res. 2025;30(1):125.
7. Talbot Z, Lee A, Boet S. Hyperbaric medicine in Canadian undergraduate medical school curriculum. Diving Hyperb Med. 2023;53(2):138-141.
8. Wang SJ, Kang P. Distribution of Hyperbaric Oxygen Chambers for Noxious Gas Disaster. Prehosp Disaster Med. 2023;38(1):127-128.

9. NHS England. Reviewing Hyperbaric Oxygen Services: Consultation Guide. Accessed date:10 March 2026. Available from: https://www.england.nhs.uk/long-read/reviewing-hyperbaric-oxygen-services-consultation-guide/.
10. Piliuk K, Tomforde S. Artificial Intelligence in Emergency Medicine. A Systematic Literature Review. Int J Med Inform. 2023;180:105274.
11. Paslı S, Şahin AS, Beşer MF, Topçuoğlu H, Yadigaroğlu M, İmamoğlu M. Assessing the precision of artificial intelligence in emergency department triage decisions: Insights from a study with ChatGPT. Am J Emerg Med. 2024;78:170-175.
12. Tomaszewski C. Carbon monoxide. In: Nelson LS, Howland MA, Lewin NA, Smith SW, Goldfrank LR, Hoffman RS, eds. Goldfrank's Toxicologic Emergencies. 11th ed. New York. McGraw-Hill, 2019:1663-1675.
13. Fisher RA, Yates F. Statistical Tables for Biological, Agricultural and Medical Research. 6th ed. London. Oliver and Boyd, 1963.
14. Rose JJ, Nouraie M, Gauthier MC, et al. Clinical outcomes and mortality impact of hyperbaric oxygen therapy in patients with carbon monoxide poisoning. Crit Care Med. 2018;46(7):e649-655.
15. Rose JJ, Wang L, Xu Q, et al. Carbon monoxide poisoning: pathogenesis, management, and future directions of therapy. Am J Respir Crit Care Med. 2017;195(5):596-606.
16. Weaver L, Hopkins R, Churchill S, Deru K. Neurological outcomes 6 years after acute carbon monoxide poisoning. Undersea Hyperb Med. 2008;35(4):241-323.
17. Haas B, Stukel TA, Gomez D, et al. The mortality benefit of direct trauma center transport in a regional trauma system: a population-based analysis. J Trauma Acute Care Surg. 2012;72(6):1510-1517.
18. Prabhakaran S, O’Neill K, Stein-Spencer L, Walter J, Alberts MJ. Prehospital triage to primary stroke centers and rate of stroke thrombolysis. JAMA Neurol. 2013;70(9):1126-1132.
19. Savage ML, Hay K, Murdoch DJ, et al. Clinical outcomes in pre-hospital activation and direct cardiac catheterisation laboratory transfer of STEMI for primary PCI. Heart Lung Circ. 2022;31(7):974-984.
20. Kang DY, Cho KJ, Kwon O, et al. Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services. Scand J Trauma Resusc Emerg Med. 2020;28:1-8.
21. Blomberg SN, Folke F, Ersbøll AK, et al. Machine learning as a supportive tool to recognize cardiac arrest in emergency calls. Resuscitation. 2019;138:322-329.
22. Kijanka R, Kawecki M, Białoń P, Szlagor M, Dudek M, Bobiński R. Three-step analysis of symptoms of carbon monoxide poisoning in pre-hospital medical response teams. Emerg Med Serv. 2022;9(1):48-54.
23. Hampson NB, Piantadosi CA, Thom SR, Weaver LK. Practice recommendations in the diagnosis, management, and prevention of carbon monoxide poisoning. Am J Respir Crit Care Med. 2012;186(11):1095-1101.
24. Uehara O, Morikawa T, Harada F, et al. Performance of ChatGPT‐3.5 and ChatGPT‐4o in the Japanese National Dental Examination. J Dent Educ. 2025;89(4):459-466.
25. OpenAI. Introducing ChatGPT. Accessed date: 31 May 2025. Available from: https://openai.com/index/hello-gpt-4o/.
26. OpenAI. Introducing OpenAI o3 and o4 mini. Accessed date: 31 May 2025. Available from: https://openai.com/index/introducing-o3-and-o4-mini/.
27. OpenAI. Introducing GPT-4.5. Accessed date: 31 May 2025. Available from: https://openai.com/index/introducing-gpt-4-5/.
28. Balel Y, Zogo A, Yıldız S, Tanyeri H. Can ChatGPT-4o provide new systematic review ideas to oral and maxillofacial surgeons? J Stomatol Oral Maxillofac Surg. 2024:101979.
29. Is EE, Menekseoglu AK. Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o. Clin Rheumatol. 2024;43(11):3507-3513.
30. Caspi R, Karp PD. An evaluation of ChatGPT and Bard (Gemini) in the context of biological knowledge retrieval. Access Microbiol. 2024;6(6):000790. v3.
31. Team G, Anil R, Borgeaud S, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:231211805. 2023.
32. AI G. Model İnformation 2.5 Pro Model Card. Accessed date: 31 May 2025. Available from: https://deepmind.google/technologies/gemini/pro/.
33. DeepSeek I. DeepSeek-R1 Release. Accessed date: 31 May 2025. Available from: https://api-docs.deepseek.com/news/news250120.