Using self-organizing map and rating scale model in examining internal structure of scales

Abstract

This study compares the psychometric properties of scales developed using Exploratory Factor Analysis (EFA), Self-Organizing Map (SOM), and Andrich's Rating Scale Model (RSM). Data for the research were collected by administering the "Statistical Attitude Scale" trial form, previously used in a separate study, to 808 individuals. First, EFA, SOM and RSM were applied to decide the number of dimensions of the scale, and to select items. Subsequently, Confirmatory Factor Analysis (CFA) was used to the forms obtained from different methods and their CFA fit indices were compared. The analysis revealed variations in the number of dimensions and item distribution across different methods. Results indicated that the form generated using SOM exhibited the highest fit indices. Furthermore, the CFA fit indices of the form created with RSM were found to be satisfactory, offering detailed insights into both items and individuals.

Keywords

• Scale development
• Internal structure
• Self-organizing map
• Rating scale modal
• Exploratory factor analysis

References

1. AERA. (2014). Standards for Educational and Psychological Testing. American Educational Research Association.
2. Anastasi, A. (1976). Psychological testing. The Macmillan Company, Collier Macmillan Limited.
3. Andrich, D. (1978a). Application of a psychometric model to ordered categories which are scored with successive integers. Applied Psychological Measurement, 2(4), 581-594.
4. Andrich, D. (1978b). A rating formulation for ordered response categories. Psychometrika, 43(4), 561-573.
5. Astel, A., Tsakovski, S., Barbieri, P., & Simeonov, V. (2007). Comparison of self-organizing maps classification approach with cluster and principal components analysis for large environmental data sets. Water Research, 41, 4566 4578. https://doi:10.1016/j.watres.2007.06.030
6. Bond, T.G., & Fox, C.M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.
7. Bramer, M. (2020). Principles of data mining. Springer.
8. Brown, T.A. (2006). Confirmatory factor analysis for applied research. The Guildford Press.

9. Büyüköztürk, Ş., Çakmak, E.K., Akgün, Ö.E., Karadeniz, Ş., & Demirel, F. (2017). Bilimsel araştırma yöntemleri [Scientific research methods]. Pegem Akademi.
10. Cantó‑Cerdán, M., Cacho‑Martínez, P., Lara‑Lacárcel, F., & García‑Muñoz, A. (2021). Rasch analysis for development and reduction of symptom questionnaire for visual dysfunctions (SQVD). Scientific Reports, 11(14855), 1 10. https://doi.org/10.1038/s41598 021 94166-9
11. Chattopadhyay, M.J., Dan, P.K., & Majumdar, S. (2011). Principal Component Analysis and Self Organizing Map for Visual Clustering of Machine-Part Cell Formation in Cellular Manufacturing System. https://arxiv.org/pdf/1201.5524.pdf
12. Crocker, L.M., & Algina, J. (2008). Introduction to classical and modern test theory. Cengage Learning.
13. Çelen, Ü. (2008). Klasik test kuramı ve madde tepki kuramı yöntemleriyle geliştirilen iki testin geçerlilik ve güvenilirliğinin karşılaştırılması. [Comparison of validity and reliability of two tests developed by classical test theory and item response theory]. Elementary Education Online, 7(3), 758-768.
14. Çevik, M., & Tabaru-Örnek, G. (2020). Comparison of MATLAB and SPSS software in the prediction of academic achievement with artificial neural networks: Modeling for elementary school students. International Online Journal of Education and Teaching, 7(4), 1689-1707.
15. Çokluk, Ö., Şekercioğlu G., & Büyüköztürk, Ş. (2012). Sosyal bilimler için çok değişkenli istatistik SPSS ve LISREL uygulamaları. [SPSS and LISREL applications of multivariate statistics for social sciences]. Pegem Akademi.
16. Das, G., Chattopadhyay, M., & Gupta, S. (2016). A comparison of self-organizing maps and principal components analysis. International Journal of Market Research, 58(6). 1-20. https://doi.org/10.2501/IJMR-2016-039 De Ayala, R.J. (2009). The theory and practice of item response theory. Guilford Press.
17. Doğan, N., & Başokçu, O. (2010). İstatistik Tutum Ölçeği İçin Uygulanan Faktör Analizi ve Aşamalı Kümeleme Analizi Sonuçlarının Karşılaştırılması. [Comparison of factor analysis and progressive cluster analysis results applied to the statistics attitude scale]. Journal of Measurement and Evaluation in Education and Psychology, 1(2), 65-71.
18. Ebel, R.L., & Frisbie, D.A. (1991). Essentials of educational measurement. Englewood Cliffs, NJ: Prentice- Hall International.
19. Eğrioğlu, E., Yolcu, U., & Baş, E. (2019). Yapay sinir ağları öngörü ve tahmin uygulamaları [Artificial neural networks prediction and prediction applications]. Nobel yayınları.
20. Embretson, S.E., & Reise, S. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
21. Eriş Hasırcı, H.M. (2019). Öz düzenlemeli haritalar yöntemi ile elde edilen yapı geçerliği kanıtlarının faktör analizi ve kümeleme analizi ile karşılaştırılması. [Comparison of construct validity proofs obtained from self-organizing maps method with the factor analysis and clustering analysis] [Unpublished doctoral dissertation]. Ankara University.
22. Erkuş, A. (2012). Psikolojide ölçme ve ölçek geliştirme-1: Temel kavramlar ve işlemler. (Measurement and Scale Development in Psychology- Part 1: Basic Concepts and Processes). Pegem Akademi.
23. Fava, J.L., & Velicer, W.F. (1992). The effects of over extraction on factor and component analysis. Multivariate Behavioral Research, 27, 387-415.
24. Ferles, C., Papanikolaou, Y., Savaidis, S.P., & Mitilineos, S.A. (2021). Deep learning self-organizing map of convolutional layers. Computer Science and Information Technology (CS & IT) Computer Science Conference, 25 32. https://doi.org/10.5121/csit.2021.110303
25. Flora, D.B., & Curran, P.J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
26. Francis, L. (2001). Neural Network Demystified. 2001 Winter Forum Ratemaking Discussion Papers and Data Management/Data Quality/Data Technology Call Papers. www.cacact.org./pubs/forum/01wforum/01wf253.pdf.
27. Galvan, D., Cremasco, H., Gomes Mantovani, A.C., Bona, E., Killner, M., & Borsato, D. (2020). Kinetic study of the transesterification reaction by artificial neural networks and parametric particle swarm optimization. Fuel 267, 117221. https://doi.org/10.1016/j.fuel.2020.117221
28. Galvan, D., Effting, L., Cremasco, H., & Conte-Junior, C.A. (2020). Can socioeconomic, health, and safety data explain the spread of COVID-19 outbreak on Brazilian Federative Units? International Journal of Environmental Research and Public Health, 17(8921), 2-16. https://doi.org/10.3390/ijerph17238921
29. Galvan, D., Effting, L., Cremasco H., & Conte-Junior, C.A. (2021). The spread of the Covid-19 outbreak in Brazil: An overview by Kohonen self-organizing map networks. Medicina, 57(235), 2-19. https://doi.org/10.3390/medicina57030235
30. Güler, N., İlhan, M., & Taşdelen Teker, G. (2018). Comparing the scale values obtained from pairwise comparison scaling method and Rasch analysis. Inonu University Journal of the Faculty of Education, 19(1), 31-48. https://doi.org/10.17679/inuefd.400386
31. Grajciarova, L., Mares, J., Dvorak P., & Prochazka, A. (2012). Biomedical image analysis using self-organizing maps. Technicka, 5, 166.
32. Grove, S.K., Burns, N., & Gray, J.R. (2012). The practice of nursing research: Appraisal, synthesis, and generation of evidence (7th Ed.). Elsevier.
33. Haykin, S. (2009). Neural networks and learning machines (3rd Edition). Pearson Education.
34. İlhan, M., & Güler, N. (2017). Likert tipi ölçeklerde klasik test kuramı ile Rasch analizinden elde edilen yetenek kestirimleri arasındaki uyumun test edilmesi. [Testing the agreement between ability estimations made through classical test theory and the ones made through Rasch analysis in Likert type scales]. Ege Journal of Education, 18(1). 244-265.
35. İlhan, M., & Güler, N. (2018). Likert tipi ölçeklerde Rasch modelinin kullanımı: olumsuz değerlendirilme korkusu ölçeği-öğrenci formu (ODKÖ-ÖF) üzerinde bir uygulama. [The use of Rasch model in Likert type scales: An application on the fear of negative evaluation scale-student form (FNE-SF)]. Trakya Journal of Education, 8(4), 756-775.
36. Karlin, O., & Karlin S. (2018). Making better tests with the Rasch measurement model. InSight: A Journal of Scholarly Teaching, 13, 76-100.
37. Kiang, M.Y., & Kumar, A. (2001). An evaluation of self-organizing map networks as a robust alternative to factor analysis in data mining applications. Information Systems Research, 12(2), 177-194.
38. Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43, 59–69.
39. Kohonen, T. (2014). MATLAB Implementations and Applications of the Self-Organizing Map. Unigrafia Oy.
40. Kozan, S. (2016). Madde sınıflamada faktör analizi, çok boyutlu ölçekleme ve kümeleme analizlerinin karşılaştırılması [Comparison of factor analysis, multidimensional scaling and cluster analysis in item classification] [Unpublished master’s thesis]. Mersin University.
41. Krishnan, S., & Idris, N. (2018). Using partial credit model to improve the quality of an instrument. International Journal of Evaluation and Research in Education, 7(4), 313-316.
42. Li, C-H. (2016). Confirmatory factor analysis with ordinal data: comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48, 936–949. https://doi.org/10.3758/s13428-015-0619-7
43. Linacre, J.M. (2012). Many facet Rasch measurement: Facets tutorial. http://www.winsteps.com/a/ftutorial2.pdf
44. Linacre, J.M. (2014). A user's guide to FACETS Rasch model computer programs. http://www.winsteps.com/a/facets-manual.pdf
45. Miljković, D. (2017). Brief review of self-organizing maps. MIPRO/CTS, 1252-1257
46. Murtagh, F., & Hernandez-Pajares, M. (1995). The Kohonen self-organizing map method: An assessment. Journal of Classification, 12(2), 165-190
47. Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3rd. ed.). McGraw-Hill.
48. Ostini, R., & Nering, M.L. (2006). Polytomous item response theory models (3rd. Ed). Sage Publications.
49. Peixoto, E.M., Zanini, D.S., & Andrade J.M. (2021). Cross-cultural adaptation and psychometric properties of the Kessler distress scale (K10): an application of the rating scale model. Psicologia: Reflexão e Crítica, 34(21), 2 10. https://doi.org/10.1186/s41155-021-00186-9
50. Sadesky, G.S. (2007). Determining the structure in test performance [Unpublished doctoral dissertation]. Alberta University.
51. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.
52. Sorrosal Forradellasa, M.T., Barberà-Mariné, M.G., Fernández Bariviera, A., & Garbajosa-Cabello, M.J. (2012). Advantages of using self-organizing maps to analyse student evaluations of teaching. Fuzzy Economic Review. https://doi.org/10.25102/fer.2012.01.03
53. Sözer, E., & Kahraman, N. (2021). Investigation of psychometric properties of Likert items with the same response categories using polytomous item response theory models. Journal of Measurement and Evaluation in Education and Psychology, 12(2), 129-146. https://doi.org/10.21031/epod.819927
54. Stambuk, A., Stambuk, N., & Konjevoda P. (2007). Application of Kohonen self- organizing maps (SOM) based clustering for the assessment of religious motivation. 29th International Conference on Information Technology Interfaces, Croatia.
55. Şencan, H. (2005). Sosyal ve davranışsal ölçümlerde güvenirlik ve geçerlik [Reliability and validity in social and behavioral measurements] (1st ed.). Seçkin Yayınları.
56. Şimşek, D. (2006). Kümeleme analizi, çok boyutlu ölçekleme, doğrulayıcı ve açıklayıcı faktör analizi ile elde edilen yapı geçerliği kanıtlarının karşılaştırılması. [A comprasion of construct validity evidences obtained through cluster analysis, multidimensional scaling, confirmatory, and exploratory factor analysis] [Unpublished master’s thesis]. Hacettepe University.
57. Takasaki, H., Kawazoe, S., Miki, T., Chiba H., & Godfrey, E. (2021). Development and validity assessment of a Japanese version of the exercise adherence rating scale in participants with musculoskeletal disorders. Health and Quality Life Outcomes, 19(169), 2-8. https://doi.org/10.1186/s12955-021-01804-x.
58. Tezbaşaran, E. (2016). Temel bileşenler analizi ve yapay sinir ağı modellerinin ölçek geliştirme sürecinde kullanılabilirliğinin incelenmesi. [An investigation on usability of principal component analysis and artificial neural network models in the process of scale development] [Unpublished doctoral dissertation]. Mersin University.
59. Uysal, M. (2015). Araştırma özyeterlik ölçeğinin psikometrik özelliklerinin klasik test kuramı ve madde tepki kuramına göre incelenmesi. [An investigation of psychometric properties of research self-efficacy scale according to classical test theory and item response theory] [Unpublished master’s thesis] Gazi University.
60. Wood, J.M., Tataryn, D.J., & Gorsuch, R.L. (1996). Effects of under- and over extraction on principal axis factor analysis with varimax rotation. Psyhological Methods, 1(4), 354-365.