Research Article
PDF EndNote BibTex RIS Cite

An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs

Year 2023, Volume 31, Issue 1, 143 - 154, 31.01.2023
https://doi.org/10.24106/kefdergi.1246458

Abstract

Purpose: The purpose of this study is to predict dropouts in two runs of the same MOOC using an explainable machine learning approach. With the explainable approach, we aim to enable the interpretation of the black-box predictive models from a pedagogical perspective and to produce actionable insights for related educational interventions. The similarity and the differences in feature importance between the predictive models were also examined. Design/Methodology/Approach: This is a quantitative study performed on a large public dataset containing activity logs in a MOOC. In total, 21 features were generated and standardized before the analysis. Multi-layer perceptron neural network was used as the black-box machine learning algorithm to build the predictive models. The model performances were evaluated using the accuracy and AUC metrics. SHAP was used to obtain explainable results about the effects of different features on students’ success or failure. Findings: According to the results, the predictive models were quite accurate, showing the capacity of the features generated in capturing student engagement. With the SHAP approach, reasons for dropouts for the whole class, as well as for specific students were identified. While mostly disengagement in assignments and course wares caused dropouts in both course runs, interaction with video (the main teaching component) showed a limited predictive power. In total six features were common strong predictors in both runs, and the remaining four features belonged to only one run. Moreover, using waterfall plots, the reasons for predictions pertaining to two randomly chosen students were explored. The results showed that dropouts might be explained by different predictions for each student, and the variables associated with dropouts might be different than the predictions conducted for the whole course. Highlights: This study illustrated the use of an explainable machine learning approach called SHAP to interpret the underlying reasons for dropout predictions. Such explainable approaches offer a promising direction for creating timely class-wide interventions as well as for providing personalized support for tailored to specific students. Moreover, this study provides strong evidence that transferring predictive models between different contexts is less like to be successful.

An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs

Year 2023, Volume 31, Issue 1, 143 - 154, 31.01.2023
https://doi.org/10.24106/kefdergi.1246458

Abstract

Purpose: The purpose of this study is to predict dropouts in two runs of the same MOOC using an explainable machine learning approach. With the explainable approach, we aim to enable the interpretation of the black-box predictive models from a pedagogical perspective and to produce actionable insights for related educational interventions. The similarity and the differences in feature importance between the predictive models were also examined. Design/Methodology/Approach: This is a quantitative study performed on a large public dataset containing activity logs in a MOOC. In total, 21 features were generated and standardized before the analysis. Multi-layer perceptron neural network was used as the black-box machine learning algorithm to build the predictive models. The model performances were evaluated using the accuracy and AUC metrics. SHAP was used to obtain explainable results about the effects of different features on students’ success or failure. Findings: According to the results, the predictive models were quite accurate, showing the capacity of the features generated in capturing student engagement. With the SHAP approach, reasons for dropouts for the whole class, as well as for specific students were identified. While mostly disengagement in assignments and course wares caused dropouts in both course runs, interaction with video (the main teaching component) showed a limited predictive power. In total six features were common strong predictors in both runs, and the remaining four features belonged to only one run. Moreover, using waterfall plots, the reasons for predictions pertaining to two randomly chosen students were explored. The results showed that dropouts might be explained by different predictions for each student, and the variables associated with dropouts might be different than the predictions conducted for the whole course. Highlights: This study illustrated the use of an explainable machine learning approach called SHAP to interpret the underlying reasons for dropout predictions. Such explainable approaches offer a promising direction for creating timely class-wide interventions as well as for providing personalized support for tailored to specific students. Moreover, this study provides strong evidence that transferring predictive models between different contexts is less like to be successful.

Details

Primary Language English
Subjects Education, Scientific Disciplines
Journal Section Research Article
Authors

Erkan ER> (Primary Author)
MIDDLE EAST TECHNICAL UNIVERSITY
Türkiye

Publication Date January 31, 2023
Published in Issue Year 2023, Volume 31, Issue 1

Cite

Bibtex @research article { kefdergi1246458, journal = {Kastamonu Eğitim Dergisi}, eissn = {2147-9844}, address = {Aktekke Mah. Kastamonu eğitim Fakültesi Kastamonu}, publisher = {Kastamonu University}, year = {2023}, volume = {31}, number = {1}, pages = {143 - 154}, doi = {10.24106/kefdergi.1246458}, title = {An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs}, key = {cite}, author = {Er, Erkan} }
APA Er, E. (2023). An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs . Kastamonu Eğitim Dergisi , 31 (1) , 143-154 . DOI: 10.24106/kefdergi.1246458
MLA Er, E. "An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs" . Kastamonu Eğitim Dergisi 31 (2023 ): 143-154 <https://dergipark.org.tr/en/pub/kefdergi/issue/75629/1246458>
Chicago Er, E. "An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs". Kastamonu Eğitim Dergisi 31 (2023 ): 143-154
RIS TY - JOUR T1 - An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs AU - ErkanEr Y1 - 2023 PY - 2023 N1 - doi: 10.24106/kefdergi.1246458 DO - 10.24106/kefdergi.1246458 T2 - Kastamonu Eğitim Dergisi JF - Journal JO - JOR SP - 143 EP - 154 VL - 31 IS - 1 SN - -2147-9844 M3 - doi: 10.24106/kefdergi.1246458 UR - https://doi.org/10.24106/kefdergi.1246458 Y2 - 2023 ER -
EndNote %0 Kastamonu Education Journal An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs %A Erkan Er %T An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs %D 2023 %J Kastamonu Eğitim Dergisi %P -2147-9844 %V 31 %N 1 %R doi: 10.24106/kefdergi.1246458 %U 10.24106/kefdergi.1246458
ISNAD Er, Erkan . "An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs". Kastamonu Eğitim Dergisi 31 / 1 (January 2023): 143-154 . https://doi.org/10.24106/kefdergi.1246458
AMA Er E. An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs. Kastamonu Education Journal. 2023; 31(1): 143-154.
Vancouver Er E. An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs. Kastamonu Eğitim Dergisi. 2023; 31(1): 143-154.
IEEE E. Er , "An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs", Kastamonu Eğitim Dergisi, vol. 31, no. 1, pp. 143-154, Jan. 2023, doi:10.24106/kefdergi.1246458

10037