Online Beceri Değerlendirme Aracının Kullanılabilirliğinin Değerlendirilmesi

Abstract

Bu çalışmada, düşük beceri seviyesine sahip yetişkinlerin okuryazarlık, sayısal ve dijital becerilerinin değerlendirilebilmesi için geliştirilen online beceri değerlendirme aracının kullanılabilirliğinin değerlendirilmesi amaçlanmıştır. Bu amaç doğrultusunda, çalışma grubunu, 18-40 yaşları arasında ve en fazla ortaokul mezunu olan 14 katılımcı oluşturmaktadır. Çalışmada, kullanıcı deneyimlerinin işe koşulduğu kullanıcı tabanlı yöntem kullanılmış ve online beceri değerlendirme aracının kullanılabilirliğinin değerlendirilmesinde, ISO 9241-11 tarafından belirlenen etkililik, verimlilik ve memnuniyet ölçütleri temel alınmıştır. Bu bağlamda, görevlerin tamamlanma oranı olarak kabul edilen etkililik ile görevlerin tamamlanma süresi olarak kabul edilen verimlilik ekran kayıtlarından elde edilen verilere göre belirlenmiştir. Memnuniyet ise araştırmacılar tarafından geliştirilen anketten elde edilen veriler doğrultusunda belirlenmiştir. Elde edilen bulgular, katılımcıların, görevlerin çoğunluğunu tamamlayabildiklerini ve sistemin kullanımı ile ilgili genel olarak memnun olduklarını göstermiştir. Bunun yanı sıra, kullanılabilirlik testi sırasında yapılan gözlemler ve ekran kayıtlarının detaylı incelenmesi sonucunda bazı kullanılabilirlik problemleri tespit edilmiştir. Bu problemler, sistemin son kullanıcılara açılmadan önce, daha kullanılabilir olması adına sistemin tasarımında yapılacak güncellemeler için yol gösterici olmuştur.

Keywords

• Kullanılabilirlik
• Online Değerlendirme Sistemi
• Beceri Değerlendirme
• Kullanıcı Tabanlı Yöntem

Usability Evaluation of the Online Skill Assessment Tool

Abstract

This study was conducted to evaluate the usability of the online skill assessment tool which was developed to evaluate the literacy, numeracy, and digital skills of adults with low skill levels. The sample consisted of 14 participants who were between the ages of 18 and 40 and who had graduated from secondary school at most. In the study, the user-based method, in which user experiences are employed, was used, and according to the effectiveness, efficiency and satisfaction criteria determined by ISO 9241-11, the usability of the online skill assessment tool was evaluated. According to the data obtained from the screen recordings, effectiveness was considered as the completion rate of the tasks, and efficiency was considered as the completion duration of the tasks. Satisfaction was also considered as the values obtained from the questionnaire data. The findings indicated that the participants were able to complete most of the tasks and were generally satisfied with the use of the system. Some usability problems were identified as a result of the observations and the detailed examination of the screen recordings. These problems have been a guide for updates to render the system more usable before the system is used by end users.

Keywords

• Usability
• Online Assessment System
• Skill Assessment
• User-based Method

References

1. Akgül, Y. (2020). Accessibility, usability, quality performance, and readability evaluation of university websites of Turkey: a comparative study of state and private universities. Universal Access in the Information Society, 1-14. doi:10.1007/s10209-020-00715-w. google scholar
2. Althobaiti, M. M., & Mayhew, P. (2015, September). Assessing the usability of learning management system: user experience study. In Second International Conference on E-Learning, E-Education, and Online Training (pp. 9-18). google scholar
3. Anam, H., Sadiq, M., & Jamil, H. (2020). Development of System Usability Scale (SUS) for the Urdu Language. International Journal of Computer Science and Information Security, 18(6), 73-78. google scholar
4. Battal, A., & Çağıltay, K. (2015). Investigation of usage frequency and department on usability ıssues with the Online Examination System (OSSI). Mersin University Journal of the Faculty of Education, 11(3), 752-763. google scholar
5. Bayrak, T., & Akcam, B. (2015). Exploring benefits of a web based testing and training tool. Procedia-Social and Behavioral Sciences, 195, 1032-1041. Doi:10.1016/j.sbspro.2015.06.146. google scholar
6. Cengiz, E. (2016). Hacettepe Üniversitesi Kütüphaneleri web sitesinin kullanılabilirliğinin eğitimle desteklenmesi ve değerlendirilmesi: Hacettepe Üniversitesi Bilgi ve Belge Yönetimi Bölümü öğrencileri üzerine bir çalışma. Türk Kütüphaneciliği, 30(1), 84-95. google scholar
7. Chang, C., & Almaghalsah, H. (2020). Usability evaluation of e-government websites: A case study from Taiwan. International Journal of Data and Network Science, 4(2), 127-138. google scholar
8. Chao, J. Y., Chao, S. J., Yao, L. Y., & Liu, C. H. (2016). A case study of design and usability evaluation of the collaborative problem solving instructional platform system. Eurasia Journal of Mathematics, Science and Technology Education, 12(10), 2647-2655. Doi:10.12973/eurasia.2016.1278a. google scholar

9. Cheon, J., & Grant, M. (2009). Active listening: Web-based assessment tool for communication and active listening skill development. TechTrends, 53(6), 24-34. Doi:10.1007/s11528-009-0340-4. google scholar
10. Çağıltay, K. (2016). İnsan bilgisayar etkileşimi ve eğitim teknolojileri. In K. Çağıltay and Y. Göktaş (Eds.) Öğretim Teknolojilerinin Temelleri: Teoriler Araştırmalar Eğilimler (pp. 297-314). Pegem Akademi, Ankara. google scholar
11. de Castro Lozano C. et al. (2011). Usable interface design for everyone. In Cipolla Ficarra FV, de Castro Lozano C, Nicol E, Kratky A, Cipolla-Ficarra M (Eds) Human-Computer Interaction, Tourism and Cultural Heritage (pp. 157-172). Springer, Berlin, Heidelberg. Doi:10.1007/978-3-642-18348-5_15. google scholar
12. Dillon, A. (2001). The evaluation of software usability. In Karwowski W (Ed) Encyclopedia of Human Factors and Ergonomics. Taylor and Francis, London. https://repository.arizona.edu/bitstream/handle/10150/105344/Adusability.pdf?sequence=1&isAllowed=y. Accessed 15 October 2020. google scholar
13. Dix, A., Finlay, J., Abowd, G., & Beale, R. (2004). Human-computer interaction. Prentice Hall, England. google scholar
14. Dumit, N. Y. (2012). Diagnostic/formative/summative assessment. https://vb.ckfu.org/attachments/%E3%E1%CA%DE%EC-%C7%E1%CE%D1%ED% CC%ED%E4/303900d1479934616-%E3%D1%CC%DA-%D4%C7%E3%E1-%E1%DF%DD%C7%ED%C7%CA-%C7%E1%DE%D3%E3-%C7%E1%DA%C7%E3-%E6%DF%DD%C7%ED%C7%CA-%C7%E1%CA%CE%D5%D5-%C7%E1%E1%DB%C9-%C7%E1%C7%E4%CC%E1%ED%D2%ED%C9-diagnostic-formative-summative-asst.pdf. Accessed 20 October 2020. google scholar
15. Ersoy, H. (2004). Bir çevrimiçi öğrenim destek sisteminin kullanılabilirlik testi: Planlama, uygulama, değerlendirme. The Turkish Online Journal of Educational Technology, 3(1), 75-82. google scholar
16. Gonçalves, R., Rocha, T., Martins, J., Branco, F., & Au-Yong-Oliveira, M. (2018). Evaluation of e-commerce websites accessibility and usability: An e-commerce platform analysis with the inclusion of blind users. Universal Access in the Information Society, 17(3), 567-583. Doi:10.1007/ s10209-017-0557-5. google scholar
17. Granic, A., & Cukusic, M. (2011). Usability testing and expert inspections complemented by educational evaluation: A case study of an e-learning platform. Journal of Educational Technology & Society, 14(2), 107-123. google scholar
18. Gülbahar, Y., Kalelioğlu, F., & Madran, O. (2008). Usability evaluation of “web macerası” as an instructional and evaluation method. Ankara Üniversitesi Eğitim Bilimleri Fakültesi Dergisi, 41(2), 209-236. google scholar
19. Güven, Ç. (1987) Okulda öğrenme-öğretme sürecinin denetimi. Ondokuz Mayıs Üniversitesi Eğitim Fakültesi Dergisi, 2(1), 271-285. google scholar
20. Harrati, N., Bouchrika, I., Tari, A., & Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computers in Human Behavior, 61, 463-471. Doi:10.1016/j.chb.2016.03.051. google scholar
21. Hasan, L. (2014). The website of the University of Jordan: Usability evaluation. International Arab Journal of e-Technology, 3(4), 258-269. google scholar
22. Hasan, L., Morris, A., & Probets, S. (2012). A comparison of usability evaluation methods for evaluating e-commerce websites. Behaviour & Information Technology, 31(7), 707-737. Doi:10.1080/0144929X.2011.596996. google scholar
23. Huang, Z., & Benyoucef, M. (2014). Usability and credibility of e-government websites. Government Information Quarterly, 31(4), 584-595. Doi:10.1016/j. giq.2014.07.002. google scholar
24. Ismail, A., & Kuppusamy, K. S. (2018). Accessibility of Indian universities’ homepages: An exploratory study. Journal of King Saud University-Computer and Information Sciences, 30(2), 268-278. google scholar
25. ISO 9241-11 (1998). Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. https://pdfs. semanticscholar.org/17e5/3f6fa806ea573e4ee0904fbb12c3a527d5f0.pdf. Accessed 10 October 2020. google scholar
26. Kakasevski, G., Mihajlov, M., Arsenovski, S., & Chungurski, S. (2008, June). Evaluating usability in learning management system Moodle. In ITI 2008 - 30th International Conference on Information Technology Interfaces (pp. 613-618). google scholar
27. KAMIS (2018). Kullanılabilirlik ilkeleri. https://kamis.gov.tr/wp-content/uploads/2018/03/Bolum3.pdf. Accessed 12 October 2020. google scholar
28. Karahoca, A., Karahoca, D., & Günoğlu, S. (2009). Web tabanlı sınav otomasyon sisteminin kullanılabilirlik analizi. In 4. Ulusal Yazılım Mühendisliği Sempozyumu (pp 183-190). google scholar
29. Lai, T. L., Chen, P. Y., & Chou, C. Y. (2017, May). A user experience study of a web-based formative assessment system. In 2017 International Conference on Applied System Innovation (pp. 899-902). google scholar
30. Lee, S., & Koubek, R. J. (2010). The effects of usability and web design attributes on user preference for e-commerce web sites. Computers in Industry, 61(4), 329-341. Doi:10.1016/j.compind.2009.12.004. google scholar
31. Li, F., & Li, Y. (2011). Usability evaluation of e-commerce on B2C websites in China. Procedia Engineering, 15, 5299-5304. Doi:10.1016/j.proeng.2011.08.982. google scholar
32. Lin, H. C. K., Hsieh, M. C., Wang, C. H., Sie, Z. Y., & Chang, S. H. (2011). Establishment and usability evaluation of an interactive AR learning system on conservation of fish. Turkish Online Journal of Educational Technology, 10(4), 181-187. google scholar
33. Lyzara, R., Purwandari, B., Zulfikar, M. F., Santoso, H. B., & Solichah, I. (2019, January). E-government usability evaluation: Insights from a systematic literature review. In Proceedings of the 2nd International Conference on Software Engineering and Information Management (pp 249-253). google scholar
34. Majzub, R. M. (2009). The development of a web based ecological assessment of school readiness (WEBEASR). Procedia-Social and Behavioral Sciences, 1(1), 2568-2572. Doi:10.1016/j.sbspro.2009.01.453. google scholar
35. Mentes, S. A., & Turan, A. H. (2012). Assessing the usability of university websites: An empirical study on Namik Kemal University. Turkish Online Journal of Educational Technology, 11(3), 61-69. google scholar
36. Newell, A. F., & Gregor, P. (2002). Design for older and disabled people-where do we go from here?. Universal Access in the Information Society, 2(1), 3-7. Doi:10.1007/s10209-002-0031-9. google scholar
37. Nielsen, J. (1994). Usability engineering. Morgan Kaufmann. google scholar
38. Ogolla, J. A. (2011). Usability evaluation: Tasks susceptible to concurrent think-aloud protocol. Dissertation, Linköping University. google scholar
39. Pant, A. (2015). Usability evaluation of an academic library website: Experience with the Central Science Library, University of Delhi. The Electronic Library, 33(5), 896-915. Doi:10.1108/EL-04-2014-0067. google scholar
40. Paul, S., & Das, S. (2019). Accessibility and usability analysis of Indian e-government websites. UniversalAccess in the Information Society, 19, 949-957. Doi:10.1007/s10209-019-00704-8. google scholar
41. Pensabe-Rodriguez, A., Lopez-Dominguez, E., Hernandez-Velazquez, Y., Dominguez-Isidro, S., & De-la-Calleja, J. (2020). Context-aware mobile learning system: Usability assessment based on a field study. Telematics and Informatics, 48, 101346. Doi: 10.1016/j.tele.2020.101346. google scholar
42. Phongphaew, N., & Jiamsanguanwong, A. (2017, July). Usability evaluation on learning management system. In International Conference on Applied Human Factors and Ergonomics (pp 39-48). google scholar
43. Sagar, K., & Saha, A. (2017). A systematic review of software usability studies. International Journal of Information Technology. Doi:10.1007/ s41870-017-0048-1. google scholar
44. Seffah, A., Donyaee, M., Kline, R. B., & Padda, H. K. (2006). Usability measurement and metrics: A consolidated model. Software Quality Journal, 14(2), 159-178. Doi:10.1007/s11219-006-7600-8. google scholar
45. Senemoğlu, N. (2013) Gelişim, öğrenme ve öğretim. Kuramdan uygulamaya (23.baskı). Yargı Yayınevi, Ankara. google scholar
46. Shi, L., Al Qudah, D., Qaffas, A., & Cristea, A. I. (2013, June). Topolor: a social personalized adaptive e-learning system. In International Conference on User Modeling, Adaptation, and Personalization (pp 338-340). google scholar
47. Story, M. F., Mueller, J. L., & Mace, R. L. (1998). The universal design file: Designing for people of all ages and abilities. https://files.eric.ed.gov/fulltext/ ED460554.pdf. Accessed 15 October 2020. google scholar
48. Şahin, İ., Hebebci, M. T., & Çelik, İ. (2014, May). Çevrimiçi öğrenme ortamlarında kullanılabilirlik: Bir literatür tarama çalışması [Usability in the online learning environments: A literature review study]. In International Conference on Education in Mathematics, Science & Technology (pp 1354-1362). google scholar
49. TÜİK (2019). Sorgulama ekranı. https://biruni.tuik.gov.tr/medas/. Accessed 11 October 2020. google scholar
50. Vairamuthu, S., & Anouncia, S. M. (2016). Reconnoitering students’ satisfaction of an online based assessment system to improve usability using PSO: An examination into a problem solving and programming course. Engineering, Technology & Applied Science Research, 6(5), 1207-1211. Doi:10.48084/ etasr.800. google scholar
51. Van Den Haak, M., De Jong, M., & Jan Schellens, P. (2003). Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22(5), 339-351. Doi: 10.1080/0044929031000. google scholar
52. Ventayen, R. J. M., Estira, K. L. A., De Guzman, M. J., Cabaluna, C. M., & Espinosa, N. N. (2018). Usability evaluation of google classroom: Basis for the adaptation of gsuite e-learning platform. Asia Pacific Journal ofEducation, Arts and Sciences, 5(1), 47-51. google scholar
53. Yıldırım, A. & Şimşek, H. (2013). Sosyal bilimlerde nitel araştırma yöntemleri (9.baskı). Seçkin Yayıncılık, Ankara. google scholar