Investigating the Validity and Reliability of the Mobile Application Rating Scale

Abstract

The purpose of this study is to perform the validity and reliability study of the Mobile Application Rating Scale (MARS) for a mobile learning system that contains e-contents in different forms to improve digital parenting competencies for adults. This scale was prepared by Stoyanov et al. (2015) and adapted into Turkish for the e-pulse application by Korkmaz and Arıkan (2021). In this study, the scale items were rearranged in the context of the study to evaluate “digital parenting”. This scale aims to evaluate the application with four sub-dimensions of “participation, functionality, aesthetics, and knowledge”, and it has 21 items. The study was carried out with 181 participants. Considering the validity results of the scale, the model with the highest validity among the single-factor model, four-factor model, and second-level confirmatory factor analysis models was the four-factor model, as in the original scale. Factor loads vary between 0.55 and 0.91. Considering the reliability results, the average variance extracted for the participation dimension was 0.69, for functionality 0.78, for aesthetics 0.75, for knowledge dimension 0.63, and the average variance extracted for the whole scale was 0.70. In addition, alpha reliability (stratified alpha reliability) was 0.92 for participation, 0.94 for functionality, 0.92 for aesthetics, 0.92 for knowledge, and 0.97 for the whole scale. In the context of these findings, the validity and reliability values of the scale were high.

Keywords

• Mobile learning system
• mobile app rating
• digital parenting
• scale adaptation
• validity
• reliability

References

1. Aloul, F., Zahidi, S., & El-Hajj, W. (2009). Multi factor authentication using mobile phones. International Journal of Mathematics and Computer Science, 4(2), 65-80.
2. Bardus, M., Awada, N., Ghandour, L. A., Fares, E. J., Gherbal, T., Al-Zanati, T., & Stoyanov, S. R. (2020). The Arabic version of the Mobile App Rating Scale: development and validation study. JMIR mHealth and uHealth, 8(3), e16956.
3. Çakır, H., & Arslan, İ. (2013). Mobil cihazlar için ders içerik paketinin geliştirilmesi [Development of course content package for mobile devices.]. Bilişim Teknolojileri Dergisi, 6(3), 24-33.
4. Costa., P. D., (2003) Towards a Services Platform for Context-Aware Applications”, Yüksek Lisans Tezi, University of Twente Telematics, Enschede, The Netherlands, 10-35
5. Epskamp, S. (2022). semPlot: Path Diagrams and Visual Analysis of Various SEM Packages' Output. R package version 1.1.5. https://CRAN.R-project.org/package=semPlot
6. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50.
7. Griggs, K., Bridges, L. M., & Rempel, H. G. (2009). Library/mobile: tips on designing and developing mobile web sites. Code4lib journal, (8).
8. Harrell, F.E. (2021). Hmisc: Harrell Miscellaneous. R package version 4.6-0. https://CRAN.R-project.org/package=Hmisc

9. Hee Ko, K. K., Kim, S. K., Lee, Y., Lee, J. Y., & Stoyanov, S. R. (2022). Validation of a Korean version of mobile app rating scale (MARS) for apps targeting disease management. Health Informatics Journal, 28(2), 14604582221091975.
10. Korkmaz, S., & Arıkan, G. (2021). e-Nabız Uygulamasını Değerlendirmek İçin Kullanılan Yeni Bir Araç: Mobil Uygulama Derecelendirme Ölçeği[A New Tool Used to Evaluate e-Nabız Application: Mobile Application Rating Scale.]. Ankara Hacı Bayram Veli Üniversitesi İktisadi ve İdari Bilimler Fakültesi Dergisi, 23(3), 625-636.
11. Mendi, O., Sari, M. K., Stoyanov, S., & Mendi, B. (2022). Development and validation of the Turkish version of the Mobile App Rating Scale–MARS-TR. International Journal of Medical Informatics, 166, 104843.
12. Messner, E. M., Terhorst, Y., Barke, A., Baumeister, H., Stoyanov, S., Hides, L., ... & Probst, T. (2020). The German version of the Mobile App Rating Scale (MARS-G): development and validation study. JMIR mHealth and uHealth, 8(3), e14479.
13. R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
14. Robitzsch, A. (2021). sirt: Supplementary Item Response Theory Models. R package version 3.11-21. https://CRAN.R-project.org/package=sirt
15. Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
16. Şimşek, M. V., Betin-Can, A., & Can, B. (2016). Android Uygulamalar İçin Yazılım Kalite Modeli [Software Quality Model for Android Applications].
17. Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015). Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth, 3(1), e3422.
18. Subramanya, S. R., & Yi, B. K. (2006). User interfaces for mobile content. Computer, 39(4), 85-87.
19. Terhorst, Y., Philippi, P., Sander, L. B., Schultchen, D., Paganini, S., Bardus, M., ... & Messner, E. M. (2020). Validation of the mobile application rating scale (MARS). Plos one, 15(11), e0241480.
20. Tierney, N., Cook, D., McBain, M., & Fay, C. (2021). naniar: Data Structures, Summaries, and Visualisations for Missing Data. R package version 0.6.1. https://CRAN.R-project.org/package=naniar
21. Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45(3), 1-67. DOI 10.18637/jss.v045.i03
22. Yamamoto, K., Ito, M., Sakata, M., Koizumi, S., Hashisako, M., Sato, M., ... & Furukawa, T. A. (2022). Development and validation of a Japanese version of the Mobile App Rating Scale (MARS). JMIR Mhealth and Uhealth.