Research Article
BibTex RIS Cite

Bir Öğrenci Bilgi Sisteminin Kullanılabilirliğinin Makine Öğrenmesi Teknikleriyle Tahmin Edilmesi

Year 2019, Volume: 2 Issue: 1, 10 - 18, 13.07.2019

Abstract

Sistem
kullanılabilirliği, bir sistemin özellikle tasarım ve test aşamalarında
odaklanılması gereken unsurlardan biridir, çünkü sistemin daha iyi hale
getirilmesi için sistem yöneticilerine geri bildirim sağlamaktadır.
Literatürde, sistem kullanılabilirliğinin değerlendirilmesi için Sistem
Kullanılabilirlik Ölçeği (System Usability Scale-SUS) altın standart yöntem
olarak yaygın şekilde kullanılmaktadır. Bunun yanı sıra günümüzde yapay zekânın
alt çalışma alanlarından biri olan makine öğrenmesi de sistem kullanılabilirliğinin
değerlendirilmesi konusunda araştırmacılara yeni ufuklar sağlamaktadır. Bu
çalışmada, bir Öğrenci Bilgi Sisteminin (ÖBS) kullanılabilirliğinin makine
öğrenmesi teknikleriyle tahmin edilmesi hedeflenmiştir. Çalışma yönteminde;
Veri Madenciliği için Çapraz Endüstri Standard Süreç Modeli (CRoss-Industry
Standard Process for Data Mining–CRISP-DM) kullanılmıştır. Analizler;
Türkiye’deki bir vakıf üniversitesine ait bir ÖBS’yi kullanan 324 öğrencinin
SUS’un Türkçe versiyonuna (SUS-TR) verdiği yanıtların bulunduğu “sus1” adlı
veri seti ile “sus1” veri setine öğrencilerin yaş, cinsiyet, öğrenim gördüğü
bölüm eklenerek oluşturulan “sus0” adlı veri setleri üzerinde
gerçekleştirilmiştir. C4.5 Karar Ağacı Algoritması, Naive Bayes Sınıflandırıcı
ve k-En Yakın Komşu Algoritması ile farklı modeller kurularak performans
değerlendirmesi yapılmıştır. %80’e %20’lik Hold-out ayrımıyla gerçekleştirilen
analizlerde en iyi performans, k-En Yakın Komşu Algoritmasıyla “sus0” veri seti
üzerinde elde edilmiştir (k=20 için doğruluk = 0.698, F-ölçütü = 0.796).

References

  • [1] ISO 9241-11, “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability”. International Organization for Standartization, 1998.
  • [2] Jordan PW. An Introduction to Usability. Padstow, UK, Taylor & Francis Ltd., 1998.
  • [3] Nielsen J. “Finding usability problems through heuristic evaluation”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, USA, 1992, pp. 373–380.
  • [4] Nielsen J. “Usability 101: Introduction to Usability”. 2012. https://www.nngroup.com/articles/usability-101-introduction-to-usability/ adresinden 3 Ekim 2018 tarihinde erişildi.
  • [5] Hornbæk K. “Current practice in measuring usability: Challenges to usability studies and research”. International Journal of Human-Computer Studies, 64(2), 79–102, 2006.
  • [6] Sweeney M, Maguire M, and Shackel B. “Evaluating user-computer interaction: a framework”. Int J Man Mach Stud, 38(4), 689–711, 1993.
  • [7] Rubin J, Chisnell D. "Handbook of usability testing: how to plan, design, and conduct effective tests". 2nd ed. Indianapolis, USA, Wiley Publishing Inc., 2008.
  • [8] Kieras D. “Model Based Evaluation”. In The Human–Computer Interaction Handbook Fundamentals, Evolving Technologies, and Emerging Applications, 2nd ed., Sears A, Jacko JA. Eds. Boca Raton, USA, CRC Press, 2007.
  • [9] Mukerjee S. “Student information systems – implementation challenges and the road ahead”. Journal of Higher Education Policy and Management, 34(1), 51–60, 2012.
  • [10] Carcary M, Long G, Remenyi D. “The Implementation of a New Student Management Information System (MIS) at an Irish Institute of Technology -- An Ex Post Evaluation of its Success”. Electronic Journal of Information Systems Evaluation, 10(1), 14, 2007.
  • [11] Karat CM. “A Business Case Approach to Usability Cost Justification for the Web”. In Cost-justifying usability: an update for an Internet age, 2nd ed., Bias RG, Mayhew DJ. Eds. San Francisco, USA, Morgan Kaufman Publishers, 2005.
  • [12] Brooke J. “SUS - A quick and dirty usability scale”. Digital Equipment Corporation, 7, 1986.
  • [13] Demirkol D. “Evaluation of student information system (SIS) in terms of user emotions, performance, and perceived usability: A Turkish university case”. Yüksek Lisans Tezi, Yeditepe Üniversitesi, İstanbul, 2018.
  • [14] Korvald C, Kim E, Reza H. “Evaluation and implementation of machine learning techniques in usability testing for web sites”. In Proceedings of the 47th Annual Midwest Instruction and Computing Symposium, Verona, WI, 1–11, 2014.
  • [15] Oztekin A. “A decision support system for usability evaluation of web-based information systems”. Expert Syst Appl, 38(3), 2110–2118, 2011.
  • [16] Oztekin A, Delen D, Turkyilmaz A, Zaim S. “A machine learning-based usability evaluation method for eLearning systems”. Decis Support Syst, 56, 63–73, 2013.
  • [17] Davis PA. “Learning usability assessment models for web sites”. Doctoral Thesis, Texas A&M University, USA, 2010.
  • [18] Horng D, Kittur A, Hong JI, Faloutsos C. “Making Sense of Large Network Data: Combining Rich User Interaction and Machine Learning”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada, 167–176, 2011.
  • [19] Maragoudakis M, Tselios NK, Fakotakis N, Avouris NM. “Improving SMS Usability Using Bayesian Networks”. In Methods and Applications of Artificial Intelligence, 2308, Vlahavas IP, Spyropoulos CD. Eds. Berlin, Springer, 179–190, 2002.
  • [20] Shearer C. “The CRISP-DM model: the new blueprint for data mining”. Journal of data warehousing, 5(4), 13–22, 2000.
  • [21] Balaban ME, Kartal E. Veri Madenciliği ve Makine Öğrenmesi Temel Algoritmaları ve R Dili ile Uygulamaları. 2nd ed. Beyoğlu, İstanbul, Çağlayan Kitabevi, 2018.
  • [22] Brooke J. “SUS - A quick and dirty usability scale”. In Usability Evaluation In Industry, Jordan PW, Thomas B, McClelland IL, Weerdmeester B. Eds. London, UK, Taylor & Francis Ltd., 189–194, 1996.
  • [23] Bangor A, Kortum P, Miller J. “Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale”. Journal of Usability Studies, 4(3), 114–123, 2009.
  • [24] Selçukcan Erol Ç. “Sağlık Bilimlerinde R ile Veri Madenciliği”. In R ile Veri Madenciliği Uygulamaları. 1st ed., Balaban ME, Kartal E. Eds. İstanbul, Çağlayan Kitabevi, 2016.
  • [25] Quinlan JR, “Constructing Decision Trees”. In C4.5 Programs for Machine Learning, California, USA, Morgan Kaufman Publishers, 17–26, 1993.
  • [26] Lim TS, Loh WY, Shih YS. “A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms”. Machine Learning, 40(3), 203–228, 2000.
  • [27] Ruggieri S. “Efficient C4.5”. IEEE Transactions on Knowledge and Data Engineering, 14(2), 438–444, 2002.
  • [28] McCallum A, Nigam K. “A Comparison of Event Models for Naive Bayes Text Classification”. AAAI-98 Workshop on Learning for Text Categorization, 752, 41–48, 1998.
  • [29] Zhang H. “The optimality of naive Bayes”. In Proceedings of 17th International Florida Artificial Intelligence Research Society Conference, Florida, USA, 562–567, 2004.
  • [30] Gou J, Xiong T, Kuang Y. “A Novel Weighted Voting for K-Nearest Neighbor Rule”. Journal of Computers, 6(5), 833–840, 2011.
  • [31] The R Foundation. “R: The R Project for Statistical Computing”. 2018. https://www.r-project.org/ adresinden 14 Kasım 2018 tarihinde erişildi.
  • [32] RStudio. “RStudio - Open source and enterprise-ready professional software for R”. RStudio, 2014. https://www.rstudio.com/ adresinden 14 Kasım 2018 tarihinde erişildi.
  • [33] Kuhn M. "caret: Classification and Regression Training". 2018.
  • [34] Acuna E, The CASTLE research group. "dprep: Data Pre-Processing and Visualization Functions for Classification". 2015.
  • [35] Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F. "e1071: Misc Functions of the Department of Statistics", Probability Theory Group (Formerly: E1071), TU Wien. 2017.
  • [36] Hornik K, Buchta C, Zeileis A. "Open-Source Machine Learning: R Meets Weka". Computational Statistics, 24(2), 225–232, 2009.
  • [37] Witten IH, Frank E. "Data Mining: Practical Machine Learning Tools and Techniques", 2nd ed. San Francisco, USA, Morgan Kaufmann, 2005.
  • [38] Dragulescu AA, Arendt C. "xlsx: Read, Write, Format Excel 2007 and Excel 97/2000/XP/2003 Files". 2018.
  • [39] Harper BD, Norman KL. “Improving user satisfaction: The questionnaire for user interaction satisfaction version 5.5”. In Proceedings of the 1st Annual Mid-Atlantic Human Factors Conference, 224–228, 1993.
  • [40] Lewis JR. “IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use”. Int J Hum-Comput Int, 7(1), 57–78, 1995.

Predicting Usability of a Student Information System by Using Machine Learning Techniques

Year 2019, Volume: 2 Issue: 1, 10 - 18, 13.07.2019

Abstract

System usability is one
of the key elements that should be focused on, especially during the design and
test phases of a system, because it provides feedback to system administrators
in order to improve the system. In the literature, System Usability Scale (SUS)
is widely used as the gold standard method to evaluate system usability. Today,
machine learning, which is one of the subfields of artificial intelligence,
also provide new perspectives on the evaluation of system usability. In this
study, it is aimed to predict usability of a Student Information System (SIS)
by using machine learning techniques. In the study method, the Cross-Industry
Standard Process for Data Mining (CRISP-DM) steps have been followed. Analysis
are performed on two different dataset namely “sus1” and “sus0”. “sus1” dataset
is consisted of demographic characteristics (age, gender, department) of 324
students using a SIS of a foundation university in Turkey, also their responses
to the Turkish version of the SUS. “sus0” includes only responses to the
Turkish version of the SUS. C4.5 Decision Tree Algorithm, Naive Bayes
Classifier and k-Nearest Neighbor Algorithm are used to create models and their
performance are evaluated. The best performance was obtained on the “sus0” data
set with 80% to 20% hold-out method (accuracy = 0.698, F-measure = 0.796 for k
= 20).

References

  • [1] ISO 9241-11, “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability”. International Organization for Standartization, 1998.
  • [2] Jordan PW. An Introduction to Usability. Padstow, UK, Taylor & Francis Ltd., 1998.
  • [3] Nielsen J. “Finding usability problems through heuristic evaluation”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, USA, 1992, pp. 373–380.
  • [4] Nielsen J. “Usability 101: Introduction to Usability”. 2012. https://www.nngroup.com/articles/usability-101-introduction-to-usability/ adresinden 3 Ekim 2018 tarihinde erişildi.
  • [5] Hornbæk K. “Current practice in measuring usability: Challenges to usability studies and research”. International Journal of Human-Computer Studies, 64(2), 79–102, 2006.
  • [6] Sweeney M, Maguire M, and Shackel B. “Evaluating user-computer interaction: a framework”. Int J Man Mach Stud, 38(4), 689–711, 1993.
  • [7] Rubin J, Chisnell D. "Handbook of usability testing: how to plan, design, and conduct effective tests". 2nd ed. Indianapolis, USA, Wiley Publishing Inc., 2008.
  • [8] Kieras D. “Model Based Evaluation”. In The Human–Computer Interaction Handbook Fundamentals, Evolving Technologies, and Emerging Applications, 2nd ed., Sears A, Jacko JA. Eds. Boca Raton, USA, CRC Press, 2007.
  • [9] Mukerjee S. “Student information systems – implementation challenges and the road ahead”. Journal of Higher Education Policy and Management, 34(1), 51–60, 2012.
  • [10] Carcary M, Long G, Remenyi D. “The Implementation of a New Student Management Information System (MIS) at an Irish Institute of Technology -- An Ex Post Evaluation of its Success”. Electronic Journal of Information Systems Evaluation, 10(1), 14, 2007.
  • [11] Karat CM. “A Business Case Approach to Usability Cost Justification for the Web”. In Cost-justifying usability: an update for an Internet age, 2nd ed., Bias RG, Mayhew DJ. Eds. San Francisco, USA, Morgan Kaufman Publishers, 2005.
  • [12] Brooke J. “SUS - A quick and dirty usability scale”. Digital Equipment Corporation, 7, 1986.
  • [13] Demirkol D. “Evaluation of student information system (SIS) in terms of user emotions, performance, and perceived usability: A Turkish university case”. Yüksek Lisans Tezi, Yeditepe Üniversitesi, İstanbul, 2018.
  • [14] Korvald C, Kim E, Reza H. “Evaluation and implementation of machine learning techniques in usability testing for web sites”. In Proceedings of the 47th Annual Midwest Instruction and Computing Symposium, Verona, WI, 1–11, 2014.
  • [15] Oztekin A. “A decision support system for usability evaluation of web-based information systems”. Expert Syst Appl, 38(3), 2110–2118, 2011.
  • [16] Oztekin A, Delen D, Turkyilmaz A, Zaim S. “A machine learning-based usability evaluation method for eLearning systems”. Decis Support Syst, 56, 63–73, 2013.
  • [17] Davis PA. “Learning usability assessment models for web sites”. Doctoral Thesis, Texas A&M University, USA, 2010.
  • [18] Horng D, Kittur A, Hong JI, Faloutsos C. “Making Sense of Large Network Data: Combining Rich User Interaction and Machine Learning”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada, 167–176, 2011.
  • [19] Maragoudakis M, Tselios NK, Fakotakis N, Avouris NM. “Improving SMS Usability Using Bayesian Networks”. In Methods and Applications of Artificial Intelligence, 2308, Vlahavas IP, Spyropoulos CD. Eds. Berlin, Springer, 179–190, 2002.
  • [20] Shearer C. “The CRISP-DM model: the new blueprint for data mining”. Journal of data warehousing, 5(4), 13–22, 2000.
  • [21] Balaban ME, Kartal E. Veri Madenciliği ve Makine Öğrenmesi Temel Algoritmaları ve R Dili ile Uygulamaları. 2nd ed. Beyoğlu, İstanbul, Çağlayan Kitabevi, 2018.
  • [22] Brooke J. “SUS - A quick and dirty usability scale”. In Usability Evaluation In Industry, Jordan PW, Thomas B, McClelland IL, Weerdmeester B. Eds. London, UK, Taylor & Francis Ltd., 189–194, 1996.
  • [23] Bangor A, Kortum P, Miller J. “Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale”. Journal of Usability Studies, 4(3), 114–123, 2009.
  • [24] Selçukcan Erol Ç. “Sağlık Bilimlerinde R ile Veri Madenciliği”. In R ile Veri Madenciliği Uygulamaları. 1st ed., Balaban ME, Kartal E. Eds. İstanbul, Çağlayan Kitabevi, 2016.
  • [25] Quinlan JR, “Constructing Decision Trees”. In C4.5 Programs for Machine Learning, California, USA, Morgan Kaufman Publishers, 17–26, 1993.
  • [26] Lim TS, Loh WY, Shih YS. “A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms”. Machine Learning, 40(3), 203–228, 2000.
  • [27] Ruggieri S. “Efficient C4.5”. IEEE Transactions on Knowledge and Data Engineering, 14(2), 438–444, 2002.
  • [28] McCallum A, Nigam K. “A Comparison of Event Models for Naive Bayes Text Classification”. AAAI-98 Workshop on Learning for Text Categorization, 752, 41–48, 1998.
  • [29] Zhang H. “The optimality of naive Bayes”. In Proceedings of 17th International Florida Artificial Intelligence Research Society Conference, Florida, USA, 562–567, 2004.
  • [30] Gou J, Xiong T, Kuang Y. “A Novel Weighted Voting for K-Nearest Neighbor Rule”. Journal of Computers, 6(5), 833–840, 2011.
  • [31] The R Foundation. “R: The R Project for Statistical Computing”. 2018. https://www.r-project.org/ adresinden 14 Kasım 2018 tarihinde erişildi.
  • [32] RStudio. “RStudio - Open source and enterprise-ready professional software for R”. RStudio, 2014. https://www.rstudio.com/ adresinden 14 Kasım 2018 tarihinde erişildi.
  • [33] Kuhn M. "caret: Classification and Regression Training". 2018.
  • [34] Acuna E, The CASTLE research group. "dprep: Data Pre-Processing and Visualization Functions for Classification". 2015.
  • [35] Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F. "e1071: Misc Functions of the Department of Statistics", Probability Theory Group (Formerly: E1071), TU Wien. 2017.
  • [36] Hornik K, Buchta C, Zeileis A. "Open-Source Machine Learning: R Meets Weka". Computational Statistics, 24(2), 225–232, 2009.
  • [37] Witten IH, Frank E. "Data Mining: Practical Machine Learning Tools and Techniques", 2nd ed. San Francisco, USA, Morgan Kaufmann, 2005.
  • [38] Dragulescu AA, Arendt C. "xlsx: Read, Write, Format Excel 2007 and Excel 97/2000/XP/2003 Files". 2018.
  • [39] Harper BD, Norman KL. “Improving user satisfaction: The questionnaire for user interaction satisfaction version 5.5”. In Proceedings of the 1st Annual Mid-Atlantic Human Factors Conference, 224–228, 1993.
  • [40] Lewis JR. “IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use”. Int J Hum-Comput Int, 7(1), 57–78, 1995.
There are 40 citations in total.

Details

Primary Language Turkish
Subjects Engineering
Journal Section Articles
Authors

Denizhan Demirkol 0000-0001-7343-9610

Elif Kartal 0000-0003-4667-1806

Çağla Şeneler This is me

Sevinç Gülseçen

Publication Date July 13, 2019
Published in Issue Year 2019 Volume: 2 Issue: 1

Cite

APA Demirkol, D., Kartal, E., Şeneler, Ç., Gülseçen, S. (2019). Bir Öğrenci Bilgi Sisteminin Kullanılabilirliğinin Makine Öğrenmesi Teknikleriyle Tahmin Edilmesi. Veri Bilimi, 2(1), 10-18.



Dergimizin Tarandığı Dizinler (İndeksler)


Academic Resource Index

logo.png

journalseeker.researchbib.com

Google Scholar

scholar_logo_64dp.png

ASOS Index

asos-index.png

Rooting Index

logo.png

www.rootindexing.com

The JournalTOCs Index

journal-tocs-logo.jpg?w=584

www.journaltocs.ac.uk

General Impact Factor (GIF) Index

images?q=tbn%3AANd9GcQ0CrEQm4bHBnwh4XJv9I3ZCdHgQarj_qLyPTkGpeoRRmNh10eC

generalif.com

Directory of Research Journals Indexing

DRJI_Logo.jpg

olddrji.lbp.world/indexedJournals.aspx

I2OR Index

8c492a0a466f9b2cd59ec89595639a5c?AccessKeyId=245B99561176BAE11FEB&disposition=0&alloworigin=1

http://www.i2or.com/8.html



logo.png