Online Beceri Değerlendirme Aracının Kullanılabilirliğinin Değerlendirilmesi

Year 2022, , 19 - 32, 28.06.2022

Merve Yıldız , Muhammet Berigel , Furkan Kalyoncu , Özlem Özgenç

https://doi.org/10.26650/acin.1077400

Abstract

Bu çalışmada, düşük beceri seviyesine sahip yetişkinlerin okuryazarlık, sayısal ve dijital becerilerinin değerlendirilebilmesi için geliştirilen online beceri değerlendirme aracının kullanılabilirliğinin değerlendirilmesi amaçlanmıştır. Bu amaç doğrultusunda, çalışma grubunu, 18-40 yaşları arasında ve en fazla ortaokul mezunu olan 14 katılımcı oluşturmaktadır. Çalışmada, kullanıcı deneyimlerinin işe koşulduğu kullanıcı tabanlı yöntem kullanılmış ve online beceri değerlendirme aracının kullanılabilirliğinin değerlendirilmesinde, ISO 9241-11 tarafından belirlenen etkililik, verimlilik ve memnuniyet ölçütleri temel alınmıştır. Bu bağlamda, görevlerin tamamlanma oranı olarak kabul edilen etkililik ile görevlerin tamamlanma süresi olarak kabul edilen verimlilik ekran kayıtlarından elde edilen verilere göre belirlenmiştir. Memnuniyet ise araştırmacılar tarafından geliştirilen anketten elde edilen veriler doğrultusunda belirlenmiştir. Elde edilen bulgular, katılımcıların, görevlerin çoğunluğunu tamamlayabildiklerini ve sistemin kullanımı ile ilgili genel olarak memnun olduklarını göstermiştir. Bunun yanı sıra, kullanılabilirlik testi sırasında yapılan gözlemler ve ekran kayıtlarının detaylı incelenmesi sonucunda bazı kullanılabilirlik problemleri tespit edilmiştir. Bu problemler, sistemin son kullanıcılara açılmadan önce, daha kullanılabilir olması adına sistemin tasarımında yapılacak güncellemeler için yol gösterici olmuştur.

Keywords

Kullanılabilirlik, Online Değerlendirme Sistemi, Beceri Değerlendirme, Kullanıcı Tabanlı Yöntem

Usability Evaluation of the Online Skill Assessment Tool

Abstract

This study was conducted to evaluate the usability of the online skill assessment tool which was developed to evaluate the literacy, numeracy, and digital skills of adults with low skill levels. The sample consisted of 14 participants who were between the ages of 18 and 40 and who had graduated from secondary school at most. In the study, the user-based method, in which user experiences are employed, was used, and according to the effectiveness, efficiency and satisfaction criteria determined by ISO 9241-11, the usability of the online skill assessment tool was evaluated. According to the data obtained from the screen recordings, effectiveness was considered as the completion rate of the tasks, and efficiency was considered as the completion duration of the tasks. Satisfaction was also considered as the values obtained from the questionnaire data. The findings indicated that the participants were able to complete most of the tasks and were generally satisfied with the use of the system. Some usability problems were identified as a result of the observations and the detailed examination of the screen recordings. These problems have been a guide for updates to render the system more usable before the system is used by end users.

Keywords

Usability, Online Assessment System, Skill Assessment, User-based Method

References

• Akgül, Y. (2020). Accessibility, usability, quality performance, and readability evaluation of university websites of Turkey: a comparative study of state and private universities. Universal Access in the Information Society, 1-14. doi:10.1007/s10209-020-00715-w. google scholar
• Althobaiti, M. M., & Mayhew, P. (2015, September). Assessing the usability of learning management system: user experience study. In Second International Conference on E-Learning, E-Education, and Online Training (pp. 9-18). google scholar
• Anam, H., Sadiq, M., & Jamil, H. (2020). Development of System Usability Scale (SUS) for the Urdu Language. International Journal of Computer Science and Information Security, 18(6), 73-78. google scholar
• Battal, A., & Çağıltay, K. (2015). Investigation of usage frequency and department on usability ıssues with the Online Examination System (OSSI). Mersin University Journal of the Faculty of Education, 11(3), 752-763. google scholar
• Bayrak, T., & Akcam, B. (2015). Exploring benefits of a web based testing and training tool. Procedia-Social and Behavioral Sciences, 195, 1032-1041. Doi:10.1016/j.sbspro.2015.06.146. google scholar
• Cengiz, E. (2016). Hacettepe Üniversitesi Kütüphaneleri web sitesinin kullanılabilirliğinin eğitimle desteklenmesi ve değerlendirilmesi: Hacettepe Üniversitesi Bilgi ve Belge Yönetimi Bölümü öğrencileri üzerine bir çalışma. Türk Kütüphaneciliği, 30(1), 84-95. google scholar
• Chang, C., & Almaghalsah, H. (2020). Usability evaluation of e-government websites: A case study from Taiwan. International Journal of Data and Network Science, 4(2), 127-138. google scholar
• Chao, J. Y., Chao, S. J., Yao, L. Y., & Liu, C. H. (2016). A case study of design and usability evaluation of the collaborative problem solving instructional platform system. Eurasia Journal of Mathematics, Science and Technology Education, 12(10), 2647-2655. Doi:10.12973/eurasia.2016.1278a. google scholar
• Cheon, J., & Grant, M. (2009). Active listening: Web-based assessment tool for communication and active listening skill development. TechTrends, 53(6), 24-34. Doi:10.1007/s11528-009-0340-4. google scholar
• Çağıltay, K. (2016). İnsan bilgisayar etkileşimi ve eğitim teknolojileri. In K. Çağıltay and Y. Göktaş (Eds.) Öğretim Teknolojilerinin Temelleri: Teoriler Araştırmalar Eğilimler (pp. 297-314). Pegem Akademi, Ankara. google scholar
• de Castro Lozano C. et al. (2011). Usable interface design for everyone. In Cipolla Ficarra FV, de Castro Lozano C, Nicol E, Kratky A, Cipolla-Ficarra M (Eds) Human-Computer Interaction, Tourism and Cultural Heritage (pp. 157-172). Springer, Berlin, Heidelberg. Doi:10.1007/978-3-642-18348-5_15. google scholar
• Dillon, A. (2001). The evaluation of software usability. In Karwowski W (Ed) Encyclopedia of Human Factors and Ergonomics. Taylor and Francis, London. https://repository.arizona.edu/bitstream/handle/10150/105344/Adusability.pdf?sequence=1&isAllowed=y. Accessed 15 October 2020. google scholar
• Dix, A., Finlay, J., Abowd, G., & Beale, R. (2004). Human-computer interaction. Prentice Hall, England. google scholar
• Dumit, N. Y. (2012). Diagnostic/formative/summative assessment. https://vb.ckfu.org/attachments/%E3%E1%CA%DE%EC-%C7%E1%CE%D1%ED% CC%ED%E4/303900d1479934616-%E3%D1%CC%DA-%D4%C7%E3%E1-%E1%DF%DD%C7%ED%C7%CA-%C7%E1%DE%D3%E3-%C7%E1%DA%C7%E3-%E6%DF%DD%C7%ED%C7%CA-%C7%E1%CA%CE%D5%D5-%C7%E1%E1%DB%C9-%C7%E1%C7%E4%CC%E1%ED%D2%ED%C9-diagnostic-formative-summative-asst.pdf. Accessed 20 October 2020. google scholar
• Ersoy, H. (2004). Bir çevrimiçi öğrenim destek sisteminin kullanılabilirlik testi: Planlama, uygulama, değerlendirme. The Turkish Online Journal of Educational Technology, 3(1), 75-82. google scholar
• Gonçalves, R., Rocha, T., Martins, J., Branco, F., & Au-Yong-Oliveira, M. (2018). Evaluation of e-commerce websites accessibility and usability: An e-commerce platform analysis with the inclusion of blind users. Universal Access in the Information Society, 17(3), 567-583. Doi:10.1007/ s10209-017-0557-5. google scholar
• Granic, A., & Cukusic, M. (2011). Usability testing and expert inspections complemented by educational evaluation: A case study of an e-learning platform. Journal of Educational Technology & Society, 14(2), 107-123. google scholar
• Gülbahar, Y., Kalelioğlu, F., & Madran, O. (2008). Usability evaluation of “web macerası” as an instructional and evaluation method. Ankara Üniversitesi Eğitim Bilimleri Fakültesi Dergisi, 41(2), 209-236. google scholar
• Güven, Ç. (1987) Okulda öğrenme-öğretme sürecinin denetimi. Ondokuz Mayıs Üniversitesi Eğitim Fakültesi Dergisi, 2(1), 271-285. google scholar
• Harrati, N., Bouchrika, I., Tari, A., & Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computers in Human Behavior, 61, 463-471. Doi:10.1016/j.chb.2016.03.051. google scholar
• Hasan, L. (2014). The website of the University of Jordan: Usability evaluation. International Arab Journal of e-Technology, 3(4), 258-269. google scholar
• Hasan, L., Morris, A., & Probets, S. (2012). A comparison of usability evaluation methods for evaluating e-commerce websites. Behaviour & Information Technology, 31(7), 707-737. Doi:10.1080/0144929X.2011.596996. google scholar
• Huang, Z., & Benyoucef, M. (2014). Usability and credibility of e-government websites. Government Information Quarterly, 31(4), 584-595. Doi:10.1016/j. giq.2014.07.002. google scholar
• Ismail, A., & Kuppusamy, K. S. (2018). Accessibility of Indian universities’ homepages: An exploratory study. Journal of King Saud University-Computer and Information Sciences, 30(2), 268-278. google scholar
• ISO 9241-11 (1998). Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. https://pdfs. semanticscholar.org/17e5/3f6fa806ea573e4ee0904fbb12c3a527d5f0.pdf. Accessed 10 October 2020. google scholar
• Kakasevski, G., Mihajlov, M., Arsenovski, S., & Chungurski, S. (2008, June). Evaluating usability in learning management system Moodle. In ITI 2008 - 30th International Conference on Information Technology Interfaces (pp. 613-618). google scholar
• KAMIS (2018). Kullanılabilirlik ilkeleri. https://kamis.gov.tr/wp-content/uploads/2018/03/Bolum3.pdf. Accessed 12 October 2020. google scholar
• Karahoca, A., Karahoca, D., & Günoğlu, S. (2009). Web tabanlı sınav otomasyon sisteminin kullanılabilirlik analizi. In 4. Ulusal Yazılım Mühendisliği Sempozyumu (pp 183-190). google scholar
• Lai, T. L., Chen, P. Y., & Chou, C. Y. (2017, May). A user experience study of a web-based formative assessment system. In 2017 International Conference on Applied System Innovation (pp. 899-902). google scholar
• Lee, S., & Koubek, R. J. (2010). The effects of usability and web design attributes on user preference for e-commerce web sites. Computers in Industry, 61(4), 329-341. Doi:10.1016/j.compind.2009.12.004. google scholar
• Li, F., & Li, Y. (2011). Usability evaluation of e-commerce on B2C websites in China. Procedia Engineering, 15, 5299-5304. Doi:10.1016/j.proeng.2011.08.982. google scholar
• Lin, H. C. K., Hsieh, M. C., Wang, C. H., Sie, Z. Y., & Chang, S. H. (2011). Establishment and usability evaluation of an interactive AR learning system on conservation of fish. Turkish Online Journal of Educational Technology, 10(4), 181-187. google scholar
• Lyzara, R., Purwandari, B., Zulfikar, M. F., Santoso, H. B., & Solichah, I. (2019, January). E-government usability evaluation: Insights from a systematic literature review. In Proceedings of the 2nd International Conference on Software Engineering and Information Management (pp 249-253). google scholar
• Majzub, R. M. (2009). The development of a web based ecological assessment of school readiness (WEBEASR). Procedia-Social and Behavioral Sciences, 1(1), 2568-2572. Doi:10.1016/j.sbspro.2009.01.453. google scholar
• Mentes, S. A., & Turan, A. H. (2012). Assessing the usability of university websites: An empirical study on Namik Kemal University. Turkish Online Journal of Educational Technology, 11(3), 61-69. google scholar
• Newell, A. F., & Gregor, P. (2002). Design for older and disabled people-where do we go from here?. Universal Access in the Information Society, 2(1), 3-7. Doi:10.1007/s10209-002-0031-9. google scholar
• Nielsen, J. (1994). Usability engineering. Morgan Kaufmann. google scholar
• Ogolla, J. A. (2011). Usability evaluation: Tasks susceptible to concurrent think-aloud protocol. Dissertation, Linköping University. google scholar
• Pant, A. (2015). Usability evaluation of an academic library website: Experience with the Central Science Library, University of Delhi. The Electronic Library, 33(5), 896-915. Doi:10.1108/EL-04-2014-0067. google scholar
• Paul, S., & Das, S. (2019). Accessibility and usability analysis of Indian e-government websites. UniversalAccess in the Information Society, 19, 949-957. Doi:10.1007/s10209-019-00704-8. google scholar
• Pensabe-Rodriguez, A., Lopez-Dominguez, E., Hernandez-Velazquez, Y., Dominguez-Isidro, S., & De-la-Calleja, J. (2020). Context-aware mobile learning system: Usability assessment based on a field study. Telematics and Informatics, 48, 101346. Doi: 10.1016/j.tele.2020.101346. google scholar
• Phongphaew, N., & Jiamsanguanwong, A. (2017, July). Usability evaluation on learning management system. In International Conference on Applied Human Factors and Ergonomics (pp 39-48). google scholar
• Sagar, K., & Saha, A. (2017). A systematic review of software usability studies. International Journal of Information Technology. Doi:10.1007/ s41870-017-0048-1. google scholar
• Seffah, A., Donyaee, M., Kline, R. B., & Padda, H. K. (2006). Usability measurement and metrics: A consolidated model. Software Quality Journal, 14(2), 159-178. Doi:10.1007/s11219-006-7600-8. google scholar
• Senemoğlu, N. (2013) Gelişim, öğrenme ve öğretim. Kuramdan uygulamaya (23.baskı). Yargı Yayınevi, Ankara. google scholar
• Shi, L., Al Qudah, D., Qaffas, A., & Cristea, A. I. (2013, June). Topolor: a social personalized adaptive e-learning system. In International Conference on User Modeling, Adaptation, and Personalization (pp 338-340). google scholar
• Story, M. F., Mueller, J. L., & Mace, R. L. (1998). The universal design file: Designing for people of all ages and abilities. https://files.eric.ed.gov/fulltext/ ED460554.pdf. Accessed 15 October 2020. google scholar
• Şahin, İ., Hebebci, M. T., & Çelik, İ. (2014, May). Çevrimiçi öğrenme ortamlarında kullanılabilirlik: Bir literatür tarama çalışması [Usability in the online learning environments: A literature review study]. In International Conference on Education in Mathematics, Science & Technology (pp 1354-1362). google scholar
• TÜİK (2019). Sorgulama ekranı. https://biruni.tuik.gov.tr/medas/. Accessed 11 October 2020. google scholar
• Vairamuthu, S., & Anouncia, S. M. (2016). Reconnoitering students’ satisfaction of an online based assessment system to improve usability using PSO: An examination into a problem solving and programming course. Engineering, Technology & Applied Science Research, 6(5), 1207-1211. Doi:10.48084/ etasr.800. google scholar
• Van Den Haak, M., De Jong, M., & Jan Schellens, P. (2003). Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22(5), 339-351. Doi: 10.1080/0044929031000. google scholar
• Ventayen, R. J. M., Estira, K. L. A., De Guzman, M. J., Cabaluna, C. M., & Espinosa, N. N. (2018). Usability evaluation of google classroom: Basis for the adaptation of gsuite e-learning platform. Asia Pacific Journal ofEducation, Arts and Sciences, 5(1), 47-51. google scholar
• Yıldırım, A. & Şimşek, H. (2013). Sosyal bilimlerde nitel araştırma yöntemleri (9.baskı). Seçkin Yayıncılık, Ankara. google scholar