Kamu Web Sitelerinin Rehber Tabanlı Kullanılabilirlik Değerlendirmelerinin Yapay Zekâ Destekli Otomasyonu

Abstract

Bu çalışmada, kamu kurumlarının internet sitelerinin kullanılabilirlik değerlendirmelerinde kullanılan Kamu Kurumları İnternet Siteleri Rehberi (KAMİS)’nin uzman değerlendirmesinin zaman alıcı olması ve öznel yargıya açık olması nedeniyle, Büyük Dil Modelleri (LLM) aracılığıyla otomatik olarak yapılabilmesi araştırılmıştır. Bu kapsamda, rehber tabanlı değerlendirmelerin insan uzmanlardan bağımsız, hızlı ve nesnel bir biçimde gerçekleştirilebilmesi hedeflenmiştir. Çalışmada altı güncel LLM, KAMİS Rehberi’nin iki başlıkta yer alan 17 maddesi üzerinde test edilmiştir. Her madde için özel komutlar hazırlanmış; modellerden “Evet/Hayır/Kısmen” yanıtları ve gerekçeleri istenmiştir. Üretilen gerekçeler BLEU, METEOR ve BERTScore ölçütleri ile benzerlik açısından karşılaştırılmıştır. Modeller 17 maddenin 9’unda ortak sonuca ulaşmış; özellikle ChatGPT 5, Gemini Flash 2.5 ve Deepseek-V3’ün yüksek tutarlılık sergilediği görülmüştür. Benzerlik analizlerinde en güvenilir yöntemin BERTScore olduğu tespit edilmiştir. Bu çalışma, kullanılabilirlik rehberlerinin LLM’lerle otomatik değerlendirilmesine yönelik ilk kapsamlı uygulamalardan biri olup, kullanılabilirlik testlerinin dakikalar içinde, insan etkisinden bağımsız ve tekrarlanabilir biçimde yapılabileceğini ortaya koymaktadır.

Keywords

• Kullanılabilirlik
• Büyük Dil Modelleri
• Kullanılabilirlik Rehberleri
• LLM Değerlendirme Ölçütleri

Supporting Institution

Bu çalışma TÜBİTAK BİLGEM Yazılım Test ve Kalite Değerlendirme Laboratuvarı'nın öz kaynakları ile gerçekleştirilmiştir.

Ethical Statement

Yazar(lar) tarafından bu çalışmada bilimsel ve etik ilkelere uyulduğu ve yararlanılan tüm çalışmaların kaynakçada belirtildiği beyan edilmiştir.

Thanks

Yazarlar, TÜBİTAK BİLGEM’de yürütülen nitelikli projelerin yapay zekâdaki güncel gelişmelere uyumu için desteklerini esirgemeyen Merkez Başkanı V. Doç. Dr. Ali GÖRÇİN’e, bu çalışmada kullanılan 4 büyük dil modeline ilave olarak TÜBİTAK BİLGEM Yazılım Teknolojileri Enstitüsü’nde geliştirilmekte olan AskAI’a dahil 2 modelin de teste dahil olmasına verdikleri katkıdan dolayı AskAI ekibine ve Proje Yürütücüsü Mehmet Emin KÜÇÜKER’e teşekkür eder.

Artificial Intelligence Supported Automation of Guideline-Based Usability Evaluations of The State-Run Websites

Abstract

In this study, the Public Institutions Internet Sites Guide (KAMIS) used in usability evaluations of public institution websites was investigated for its potential to be automatically handled using Large Language Models (LLMs) due to the time-consuming nature and subjectivity of experts. Within this scope, the goal was to enable guide-based evaluations to be performed independently of human experts in a fast and objective manner. In the study, six state-of-the-art LLMs were tested on 17 items under two headings in the KAMIS Guide. Specific commands were prepared for each item; the models were asked to provide “Yes/No/Partially” answers and justifications. The generated justifications were compared in terms of similarity using the BLEU, METEOR, and BERTScore metrics. The models reached a common result in 9 out of 17 items; especially ChatGPT 5, Gemini Flash 2.5 and Deepseek-V3 showed high consistency. BERTScore was found to be the most reliable method in similarity analyses. This study is one of the first comprehensive applications of automated evaluation of usability guidelines with LLMs, demonstrating that usability testing can be done in minutes, independent of human influence, and repeatable.

Keywords

• Kullanılabilirlik
• Büyük Dil Modelleri
• Kullanılabilirlik Rehberleri
• LLM Değerlendirme Ölçütleri

Supporting Institution

This study was conducted using the internal resources of the TÜBİTAK BİLGEM Software Testing and Quality Assessment Laboratory.

Ethical Statement

It was declared by the author(s) that scientific and ethical principles have been followed in this study and all the sources used have been properly cited.

Thanks

The authors would like to thank TÜBİTAK BİLGEM Director Assoc. Prof. Dr. Ali Görçin , as well as the AskAI team and Project Manager Mehmet Emin KÜÇÜKER for their contribution to including two models under development at TÜBİTAK BİLGEM Software Technologies Institute, in addition to the four large language models used in this study.

References

1. Bemers-Lee T, Cailliau R. Proposal for a HyperText Project 1990.
2. Bemers-Lee T, Cailliau R. Proposal for a HyperText Project 1990.
3. Matera M, Rizzo F, Carughi GT. Web Usability: Principles and Evaluation Methods. In: Mendes E, Mosley N, editors. Web Engineering, Berlin, Heidelberg: Springer Berlin Heidelberg; 2006, p. 143–80. https://doi.org/10.1007/3-540-28218-1_5.
4. Matera M, Rizzo F, Carughi GT. Web Usability: Principles and Evaluation Methods. In: Mendes E, Mosley N, editors. Web Engineering, Berlin, Heidelberg: Springer Berlin Heidelberg; 2006, p. 143–80. https://doi.org/10.1007/3-540-28218-1_5.
5. Çağıltay K, Alacam O, Ocak N, et al. Developing ISO 9241-151 Product Certification Process: Challenges. In: Marcus A, editor. Design, User Experience, and Usability. Web, Mobile, and Product Design, vol. 8015, Berlin, Heidelberg: Springer Berlin Heidelberg; 2013, p. 334–41. https://doi.org/10.1007/978-3-642-39253-5_36.
6. Çağıltay K, Alacam O, Ocak N, et al. Developing ISO 9241-151 Product Certification Process: Challenges. In: Marcus A, editor. Design, User Experience, and Usability. Web, Mobile, and Product Design, vol. 8015, Berlin, Heidelberg: Springer Berlin Heidelberg; 2013, p. 334–41. https://doi.org/10.1007/978-3-642-39253-5_36.
7. Bevan N, Spinhof L. Are Guidelines and Standards for Web Usability Comprehensive? In: Jacko JA, editor. Human-Computer Interaction. Interaction Design and Usability, vol. 4550, Berlin, Heidelberg: Springer Berlin Heidelberg; 2007, p. 407–19. https://doi.org/10.1007/978-3-540-73105-4_45.
8. Bevan N, Spinhof L. Are Guidelines and Standards for Web Usability Comprehensive? In: Jacko JA, editor. Human-Computer Interaction. Interaction Design and Usability, vol. 4550, Berlin, Heidelberg: Springer Berlin Heidelberg; 2007, p. 407–19. https://doi.org/10.1007/978-3-540-73105-4_45.

9. Arsoy S, Kalipsiz O, Öztürk S. E-Devlet Web Siteleri İçin Rehber Tabanlı Kullanılabilirlik Değerlendirmesi. Proceedings of the 7th Turkish National Software Engineering Symposium, İzmir, Turkiye: 2013.
10. Arsoy S, Kalipsiz O, Öztürk S. E-Devlet Web Siteleri İçin Rehber Tabanlı Kullanılabilirlik Değerlendirmesi. Proceedings of the 7th Turkish National Software Engineering Symposium, İzmir, Turkiye: 2013.
11. Wang L, Ma C, Feng X, et al. A survey on large language model based autonomous agents. Front Comput Sci 2024;18:186345. https://doi.org/10.1007/s11704-024-40231-1.
12. Wang L, Ma C, Feng X, et al. A survey on large language model based autonomous agents. Front Comput Sci 2024;18:186345. https://doi.org/10.1007/s11704-024-40231-1.
13. Stige Å, Zamani ED, Mikalef P, et al. Artificial intelligence (AI) for user experience (UX) design: a systematic literature review and future research agenda. ITP 2024;37:2324–52. https://doi.org/10.1108/ITP-07-2022-0519.
14. Stige Å, Zamani ED, Mikalef P, et al. Artificial intelligence (AI) for user experience (UX) design: a systematic literature review and future research agenda. ITP 2024;37:2324–52. https://doi.org/10.1108/ITP-07-2022-0519.
15. Wallach DP, Flohr LA, Kaltenhauser A. Beyond the Buzzwords: On the Perspective of AI in UX and Vice Versa. In: Degen H, Reinerman-Jones L, editors. Artificial Intelligence in HCI, vol. 12217, Cham: Springer International Publishing; 2020, p. 146–66. https://doi.org/10.1007/978-3-030-50334-5_10.
16. Wallach DP, Flohr LA, Kaltenhauser A. Beyond the Buzzwords: On the Perspective of AI in UX and Vice Versa. In: Degen H, Reinerman-Jones L, editors. Artificial Intelligence in HCI, vol. 12217, Cham: Springer International Publishing; 2020, p. 146–66. https://doi.org/10.1007/978-3-030-50334-5_10.
17. Vanderdonckt J, Beirekdar A, Noirhomme-Fraiture M. Automated Evaluation of Web Usability and Accessibility by Guideline Review. In: Koch N, Fraternali P, Wirsing M, editors. Web Engineering, vol. 3140, Berlin, Heidelberg: Springer Berlin Heidelberg; 2004, p. 17–30. https://doi.org/10.1007/978-3-540-27834-4_4.
18. Vanderdonckt J, Beirekdar A, Noirhomme-Fraiture M. Automated Evaluation of Web Usability and Accessibility by Guideline Review. In: Koch N, Fraternali P, Wirsing M, editors. Web Engineering, vol. 3140, Berlin, Heidelberg: Springer Berlin Heidelberg; 2004, p. 17–30. https://doi.org/10.1007/978-3-540-27834-4_4.
19. Aytutuldu I, Yol O, Akgul YS. Integrating LLMs For Grading And Appeal Resolution In Computer Science Education 2025. https://doi.org/10.48550/arXiv.2504.13557.
20. Aytutuldu I, Yol O, Akgul YS. Integrating LLMs For Grading And Appeal Resolution In Computer Science Education 2025. https://doi.org/10.48550/arXiv.2504.13557.
21. Calvano M, Curci A, Lanzilotti R, et al. Leveraging Large Language Models for Usability Testing: a Preliminary Study. Companion Proceedings of the 30th International Conference on Intelligent User Interfaces, Cagliari Italy: ACM; 2025, p. 78–81. https://doi.org/10.1145/3708557.3716341.
22. Calvano M, Curci A, Lanzilotti R, et al. Leveraging Large Language Models for Usability Testing: a Preliminary Study. Companion Proceedings of the 30th International Conference on Intelligent User Interfaces, Cagliari Italy: ACM; 2025, p. 78–81. https://doi.org/10.1145/3708557.3716341.
23. Lu Y, Yao B, Gu H, et al. UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design. Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Yokohama Japan: ACM; 2025, p. 1–12. https://doi.org/10.1145/3706599.3719729.
24. Lu Y, Yao B, Gu H, et al. UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design. Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Yokohama Japan: ACM; 2025, p. 1–12. https://doi.org/10.1145/3706599.3719729.
25. Esposito A, Desolda G, Lanzilotti R, et al. SERENE: a Web platform for the UX semi-automatic evaluation of website. Proceedings of the 2022 International Conference on Advanced Visual Interfaces, Frascati, Rome Italy: ACM; 2022, p. 1–3. https://doi.org/10.1145/3531073.3534464.
26. Esposito A, Desolda G, Lanzilotti R, et al. SERENE: a Web platform for the UX semi-automatic evaluation of website. Proceedings of the 2022 International Conference on Advanced Visual Interfaces, Frascati, Rome Italy: ACM; 2022, p. 1–3. https://doi.org/10.1145/3531073.3534464.
27. Hsueh N-L, Lin H-J, Lai L-C. Applying Large Language Model to User Experience Testing. Electronics 2024;13:4633. https://doi.org/10.3390/electronics13234633.
28. Hsueh N-L, Lin H-J, Lai L-C. Applying Large Language Model to User Experience Testing. Electronics 2024;13:4633. https://doi.org/10.3390/electronics13234633.
29. Duan P, Warner J, Li Y, et al. Generating Automatic Feedback on UI Mockups with Large Language Models. Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu HI USA: ACM; 2024, p. 1–20. https://doi.org/10.1145/3613904.3642782.
30. Duan P, Warner J, Li Y, et al. Generating Automatic Feedback on UI Mockups with Large Language Models. Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu HI USA: ACM; 2024, p. 1–20. https://doi.org/10.1145/3613904.3642782.
31. Turbeville K, Muengtaweepongsa J, Stevens S, et al. LLM-powered Multimodal Insight Summarization for UX Testing. International Conference on Multimodel Interaction, San Jose Costa Rica: ACM; 2024, p. 4–11. https://doi.org/10.1145/3678957.3685701.
32. Turbeville K, Muengtaweepongsa J, Stevens S, et al. LLM-powered Multimodal Insight Summarization for UX Testing. International Conference on Multimodel Interaction, San Jose Costa Rica: ACM; 2024, p. 4–11. https://doi.org/10.1145/3678957.3685701.
33. Kuric E, Demcak P, Krajcovic M. Unmoderated Usability Studies Evolved: Can GPT Ask Useful Follow-up Questions? International Journal of Human–Computer Interaction 2024:1–18. https://doi.org/10.1080/10447318.2024.2427978.
34. Kuric E, Demcak P, Krajcovic M. Unmoderated Usability Studies Evolved: Can GPT Ask Useful Follow-up Questions? International Journal of Human–Computer Interaction 2024:1–18. https://doi.org/10.1080/10447318.2024.2427978.
35. Kuric E, Demcak P, Krajcovic M, et al. Systematic Literature Review of Automation and Artificial Intelligence in Usability Issue Detection 2025. https://doi.org/10.48550/arXiv.2504.01415.
36. Kuric E, Demcak P, Krajcovic M, et al. Systematic Literature Review of Automation and Artificial Intelligence in Usability Issue Detection 2025. https://doi.org/10.48550/arXiv.2504.01415.
37. Lombardi S, Montague E. The Role of Artificial Intelligence in Usability Testing and Its Potential Impact on Designing Inclusive Medical Devices and Digital Health Tools. Current Opinion in Biomedical Engineering 2026:100655. https://doi.org/10.1016/j.cobme.2026.100655.
38. Lombardi S, Montague E. The Role of Artificial Intelligence in Usability Testing and Its Potential Impact on Designing Inclusive Medical Devices and Digital Health Tools. Current Opinion in Biomedical Engineering 2026:100655. https://doi.org/10.1016/j.cobme.2026.100655.
39. Ha E, Choon-Kon-Yune I, Murray L, et al. Evaluating the Usability, Technical Performance, and Accuracy of Artificial Intelligence Scribes for Primary Care: Competitive Analysis. JMIR Hum Factors 2025;12:e71434–e71434. https://doi.org/10.2196/71434.
40. Ha E, Choon-Kon-Yune I, Murray L, et al. Evaluating the Usability, Technical Performance, and Accuracy of Artificial Intelligence Scribes for Primary Care: Competitive Analysis. JMIR Hum Factors 2025;12:e71434–e71434. https://doi.org/10.2196/71434.
41. Saadany H, Orăsan C. BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text. Proceedings of the Translation and Interpreting Technology Online Conference TRITON 2021, INCOMA Ltd. Shoumen, BULGARIA; 2021, p. 48–56. https://doi.org/10.26615/978-954-452-071-7_006.
42. Saadany H, Orăsan C. BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text. Proceedings of the Translation and Interpreting Technology Online Conference TRITON 2021, INCOMA Ltd. Shoumen, BULGARIA; 2021, p. 48–56. https://doi.org/10.26615/978-954-452-071-7_006.
43. Gunawan LS, Feivel B, Shiddiqi HA, et al. Evaluating the Accuracy and Naturalness of AI-Generated Translations Using BLEU and METEOR: A Comparison of ChatGPT, Gemini, Copilot, and DeepSeek. Procedia Computer Science 2025;269:815–24. https://doi.org/10.1016/j.procs.2025.09.024.
44. Gunawan LS, Feivel B, Shiddiqi HA, et al. Evaluating the Accuracy and Naturalness of AI-Generated Translations Using BLEU and METEOR: A Comparison of ChatGPT, Gemini, Copilot, and DeepSeek. Procedia Computer Science 2025;269:815–24. https://doi.org/10.1016/j.procs.2025.09.024.
45. Bommasani R, Liang P, Lee T. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences 2023;1525:140–6. https://doi.org/10.1111/nyas.15007.
46. Bommasani R, Liang P, Lee T. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences 2023;1525:140–6. https://doi.org/10.1111/nyas.15007.
47. Wang X, Wei J, Schuurmans D, et al. SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS 2023.
48. Wang X, Wei J, Schuurmans D, et al. SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS 2023.
49. Zheng L, Chiang W-L, Sheng Y, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NIPS ’23: Proceedings of the 37th International Conference on Neural Information Processing Systems, NY: ACM Press; 2023, p. 46595–623.
50. Zheng L, Chiang W-L, Sheng Y, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NIPS ’23: Proceedings of the 37th International Conference on Neural Information Processing Systems, NY: ACM Press; 2023, p. 46595–623.
51. Wei J, Wang X, Schuurmans D, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NIPS’22: Proceedings of the 36th International Conference on Neural Information Processing Systems, ACM Press; 2023.
52. Wei J, Wang X, Schuurmans D, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NIPS’22: Proceedings of the 36th International Conference on Neural Information Processing Systems, ACM Press; 2023.