Bir Öğrenci Bilgi Sisteminin Kullanılabilirliğinin Makine Öğrenmesi Teknikleriyle Tahmin Edilmesi

Year 2019, Volume 2, Issue 1, 10 - 18, 13.07.2019

Denizhan DEMİRKOL Elif KARTAL Çağla ŞENELER Sevinç GÜLSEÇEN

Abstract

Sistem kullanılabilirliği, bir sistemin özellikle tasarım ve test aşamalarında odaklanılması gereken unsurlardan biridir, çünkü sistemin daha iyi hale getirilmesi için sistem yöneticilerine geri bildirim sağlamaktadır. Literatürde, sistem kullanılabilirliğinin değerlendirilmesi için Sistem Kullanılabilirlik Ölçeği (System Usability Scale-SUS) altın standart yöntem olarak yaygın şekilde kullanılmaktadır. Bunun yanı sıra günümüzde yapay zekânın alt çalışma alanlarından biri olan makine öğrenmesi de sistem kullanılabilirliğinin değerlendirilmesi konusunda araştırmacılara yeni ufuklar sağlamaktadır. Bu çalışmada, bir Öğrenci Bilgi Sisteminin (ÖBS) kullanılabilirliğinin makine öğrenmesi teknikleriyle tahmin edilmesi hedeflenmiştir. Çalışma yönteminde; Veri Madenciliği için Çapraz Endüstri Standard Süreç Modeli (CRoss-Industry Standard Process for Data Mining–CRISP-DM) kullanılmıştır. Analizler; Türkiye’deki bir vakıf üniversitesine ait bir ÖBS’yi kullanan 324 öğrencinin SUS’un Türkçe versiyonuna (SUS-TR) verdiği yanıtların bulunduğu “sus1” adlı veri seti ile “sus1” veri setine öğrencilerin yaş, cinsiyet, öğrenim gördüğü bölüm eklenerek oluşturulan “sus0” adlı veri setleri üzerinde gerçekleştirilmiştir. C4.5 Karar Ağacı Algoritması, Naive Bayes Sınıflandırıcı ve k-En Yakın Komşu Algoritması ile farklı modeller kurularak performans değerlendirmesi yapılmıştır. %80’e %20’lik Hold-out ayrımıyla gerçekleştirilen analizlerde en iyi performans, k-En Yakın Komşu Algoritmasıyla “sus0” veri seti üzerinde elde edilmiştir (k=20 için doğruluk = 0.698, F-ölçütü = 0.796).

Keywords

danışmanlı öğrenme, kullanılabilirlik, makine öğrenmesi, öğrenci bilgi sistemi, sistem kullanılabilirlik ölçeği

Predicting Usability of a Student Information System by Using Machine Learning Techniques

Abstract

System usability is one of the key elements that should be focused on, especially during the design and test phases of a system, because it provides feedback to system administrators in order to improve the system. In the literature, System Usability Scale (SUS) is widely used as the gold standard method to evaluate system usability. Today, machine learning, which is one of the subfields of artificial intelligence, also provide new perspectives on the evaluation of system usability. In this study, it is aimed to predict usability of a Student Information System (SIS) by using machine learning techniques. In the study method, the Cross-Industry Standard Process for Data Mining (CRISP-DM) steps have been followed. Analysis are performed on two different dataset namely “sus1” and “sus0”. “sus1” dataset is consisted of demographic characteristics (age, gender, department) of 324 students using a SIS of a foundation university in Turkey, also their responses to the Turkish version of the SUS. “sus0” includes only responses to the Turkish version of the SUS. C4.5 Decision Tree Algorithm, Naive Bayes Classifier and k-Nearest Neighbor Algorithm are used to create models and their performance are evaluated. The best performance was obtained on the “sus0” data set with 80% to 20% hold-out method (accuracy = 0.698, F-measure = 0.796 for k = 20).

Keywords

supervised learning, usability, machine learning, student information systems, system usability scale

References

• [1] ISO 9241-11, “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability”. International Organization for Standartization, 1998.
• [2] Jordan PW. An Introduction to Usability. Padstow, UK, Taylor & Francis Ltd., 1998.
• [3] Nielsen J. “Finding usability problems through heuristic evaluation”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, USA, 1992, pp. 373–380.
• [4] Nielsen J. “Usability 101: Introduction to Usability”. 2012. https://www.nngroup.com/articles/usability-101-introduction-to-usability/ adresinden 3 Ekim 2018 tarihinde erişildi.
• [5] Hornbæk K. “Current practice in measuring usability: Challenges to usability studies and research”. International Journal of Human-Computer Studies, 64(2), 79–102, 2006.
• [6] Sweeney M, Maguire M, and Shackel B. “Evaluating user-computer interaction: a framework”. Int J Man Mach Stud, 38(4), 689–711, 1993.
• [7] Rubin J, Chisnell D. "Handbook of usability testing: how to plan, design, and conduct effective tests". 2nd ed. Indianapolis, USA, Wiley Publishing Inc., 2008.
• [8] Kieras D. “Model Based Evaluation”. In The Human–Computer Interaction Handbook Fundamentals, Evolving Technologies, and Emerging Applications, 2nd ed., Sears A, Jacko JA. Eds. Boca Raton, USA, CRC Press, 2007.
• [9] Mukerjee S. “Student information systems – implementation challenges and the road ahead”. Journal of Higher Education Policy and Management, 34(1), 51–60, 2012.
• [10] Carcary M, Long G, Remenyi D. “The Implementation of a New Student Management Information System (MIS) at an Irish Institute of Technology -- An Ex Post Evaluation of its Success”. Electronic Journal of Information Systems Evaluation, 10(1), 14, 2007.
• [11] Karat CM. “A Business Case Approach to Usability Cost Justification for the Web”. In Cost-justifying usability: an update for an Internet age, 2nd ed., Bias RG, Mayhew DJ. Eds. San Francisco, USA, Morgan Kaufman Publishers, 2005.
• [12] Brooke J. “SUS - A quick and dirty usability scale”. Digital Equipment Corporation, 7, 1986.
• [13] Demirkol D. “Evaluation of student information system (SIS) in terms of user emotions, performance, and perceived usability: A Turkish university case”. Yüksek Lisans Tezi, Yeditepe Üniversitesi, İstanbul, 2018.
• [14] Korvald C, Kim E, Reza H. “Evaluation and implementation of machine learning techniques in usability testing for web sites”. In Proceedings of the 47th Annual Midwest Instruction and Computing Symposium, Verona, WI, 1–11, 2014.
• [15] Oztekin A. “A decision support system for usability evaluation of web-based information systems”. Expert Syst Appl, 38(3), 2110–2118, 2011.
• [16] Oztekin A, Delen D, Turkyilmaz A, Zaim S. “A machine learning-based usability evaluation method for eLearning systems”. Decis Support Syst, 56, 63–73, 2013.
• [17] Davis PA. “Learning usability assessment models for web sites”. Doctoral Thesis, Texas A&M University, USA, 2010.
• [18] Horng D, Kittur A, Hong JI, Faloutsos C. “Making Sense of Large Network Data: Combining Rich User Interaction and Machine Learning”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada, 167–176, 2011.
• [19] Maragoudakis M, Tselios NK, Fakotakis N, Avouris NM. “Improving SMS Usability Using Bayesian Networks”. In Methods and Applications of Artificial Intelligence, 2308, Vlahavas IP, Spyropoulos CD. Eds. Berlin, Springer, 179–190, 2002.
• [20] Shearer C. “The CRISP-DM model: the new blueprint for data mining”. Journal of data warehousing, 5(4), 13–22, 2000.
• [21] Balaban ME, Kartal E. Veri Madenciliği ve Makine Öğrenmesi Temel Algoritmaları ve R Dili ile Uygulamaları. 2nd ed. Beyoğlu, İstanbul, Çağlayan Kitabevi, 2018.
• [22] Brooke J. “SUS - A quick and dirty usability scale”. In Usability Evaluation In Industry, Jordan PW, Thomas B, McClelland IL, Weerdmeester B. Eds. London, UK, Taylor & Francis Ltd., 189–194, 1996.
• [23] Bangor A, Kortum P, Miller J. “Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale”. Journal of Usability Studies, 4(3), 114–123, 2009.
• [24] Selçukcan Erol Ç. “Sağlık Bilimlerinde R ile Veri Madenciliği”. In R ile Veri Madenciliği Uygulamaları. 1st ed., Balaban ME, Kartal E. Eds. İstanbul, Çağlayan Kitabevi, 2016.
• [25] Quinlan JR, “Constructing Decision Trees”. In C4.5 Programs for Machine Learning, California, USA, Morgan Kaufman Publishers, 17–26, 1993.
• [26] Lim TS, Loh WY, Shih YS. “A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms”. Machine Learning, 40(3), 203–228, 2000.
• [27] Ruggieri S. “Efficient C4.5”. IEEE Transactions on Knowledge and Data Engineering, 14(2), 438–444, 2002.
• [28] McCallum A, Nigam K. “A Comparison of Event Models for Naive Bayes Text Classification”. AAAI-98 Workshop on Learning for Text Categorization, 752, 41–48, 1998.
• [29] Zhang H. “The optimality of naive Bayes”. In Proceedings of 17th International Florida Artificial Intelligence Research Society Conference, Florida, USA, 562–567, 2004.
• [30] Gou J, Xiong T, Kuang Y. “A Novel Weighted Voting for K-Nearest Neighbor Rule”. Journal of Computers, 6(5), 833–840, 2011.
• [31] The R Foundation. “R: The R Project for Statistical Computing”. 2018. https://www.r-project.org/ adresinden 14 Kasım 2018 tarihinde erişildi.
• [32] RStudio. “RStudio - Open source and enterprise-ready professional software for R”. RStudio, 2014. https://www.rstudio.com/ adresinden 14 Kasım 2018 tarihinde erişildi.
• [33] Kuhn M. "caret: Classification and Regression Training". 2018.
• [34] Acuna E, The CASTLE research group. "dprep: Data Pre-Processing and Visualization Functions for Classification". 2015.
• [35] Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F. "e1071: Misc Functions of the Department of Statistics", Probability Theory Group (Formerly: E1071), TU Wien. 2017.
• [36] Hornik K, Buchta C, Zeileis A. "Open-Source Machine Learning: R Meets Weka". Computational Statistics, 24(2), 225–232, 2009.
• [37] Witten IH, Frank E. "Data Mining: Practical Machine Learning Tools and Techniques", 2nd ed. San Francisco, USA, Morgan Kaufmann, 2005.
• [38] Dragulescu AA, Arendt C. "xlsx: Read, Write, Format Excel 2007 and Excel 97/2000/XP/2003 Files". 2018.
• [39] Harper BD, Norman KL. “Improving user satisfaction: The questionnaire for user interaction satisfaction version 5.5”. In Proceedings of the 1st Annual Mid-Atlantic Human Factors Conference, 224–228, 1993.
• [40] Lewis JR. “IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use”. Int J Hum-Comput Int, 7(1), 57–78, 1995.