Investigation of education value perception scale's psychometric properties according to CTT and IRT

Abstract

The purpose of this study is to develop Education Value Perception Scale (EVPS) based on Bronfenbrenner's Ecological Theory and to investigate its psychometric properties according to Classical Test Theory (CTT) and Item Response Theory (IRT). The data were collected from 2872 secondary school students by stratified purposeful sampling method. Measurement invariance of EVPS was tested by multigroup confirmatory factor analysis based on gender, and scalar invariance was observed to have been provided. The estimations based on IRT were conducted based on Graded Response Model. While high positive correlations were found between the item discriminations estimated according to different test theories, high negative correlations were identified between item means. McDonald’s Omega was calculated to be .79 according to CTT from reliability estimation methods, marginal reliability coefficient was determined to be .77 according to IRT. In the test-retest applications performed at 20-day intervals, the stability coefficient was found to be.81.

Keywords

• Education value
• Bronfenbrenners’s ecological theory
• Measurement invariance
• Classical test theory
• Item Response Theory

References

1. Akbaş, U., & Koğar, H. (2020). Nicel araştırmalarda kayıp veriler ve uç değerler: çözüm önerileri ve SPSS uygulamaları [Missing data and outliers in quantitative research; solution suggestions and SPSS applications]. Pegem Akademi.
2. Aliyev, R., Akbaş, U., & Özbay, Y. (2021). Mediating role of internal factors in predicting academic resilience. International Journal of School & Educational Psychology, 3(29), 1-16. https://doi.org/10.1080/21683603.2021.1904068
3. Arı, R. (2008). Eğitim psikolojisi [Educational psychology]. Nobel.
4. Arıcak, O.T., Avcu, A., Topçu, F., & Tutlu, M.G. (2020). Use of item response theory to validate cyberbullying sensibility scale for university students. International Journal of Assessment Tools in Educaiton, 7(1), 18-29. https://doi.org/10.21449/ijate.629584
5. Arıcı, İ. (2007). The effective factors on the students in the religious culture and ethics course [Doctoral dissertation]. Hacettepe University, Ankara. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
6. Aslantürk, İ. (2018). Ecological system theory development of a basic school safety measurement [Master’s dissertation] Ahi Evran Universtiy, Kırşehir. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
7. Ayre, C., & Scally, A.J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79-86. https://doi.org/10.1177/0748175613513808
8. Baker, F.B. (2001). The basic of item response theory. ERIC.

9. Başusta, N.B., & Gelbal, S. (2015). Gruplararası karşılaştırmarda ölçme değişmezliğinin test edilmesi: PISA öğrenci anketi örneği [Testing measurement invariance in comparisons between groups: Pisa student survey sample]. Hacettepe U. Journal of Education, 30(4), 80-90.
10. Bronfenbrenner, U. (1977). Toward experimental ecology of human development. American Psychogist, 32(7), 513-531. https://doi.org/10.1037/0003-066X.32.7.513
11. Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and desing. Harvard Universty Press.
12. Bronfenbrenner, U. (1986). Ecology of the family as a context for human development: Research perspectives. Developmental Psychology, 22(6), 723 742. https://doi.org/10.1037/0012-1649.22.6.723
13. Bronfenbrenner, U. (1994). Ecological models of human development. International Encyclopedia of Education, 3(2), 37-43.
14. Bronfenbrenner, U., & Ceci, S.J. (2004). Nature-nurture reconceptualized in developmental perspective: A bioecological model. Psychological Review, 101(4), 568-586. https://doi.org/10.1037/0033-295X.101.4.568
15. Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. K. A. Bolen and J. S. Long. (Ed), Testing Sructural Equation Models. SAGE Puplications.
16. Büyüköztürk, Ş., Kılıç Çakmak, E., Akgün, Ö. E., Karadeniz, Ş., & Demirel, F. (2018). Bilimsel araştırma yöntemleri [Scientific research methods]. Pegem Akademi.
17. Büyüköztürk, Ş. (2018). Sosyal bilimler için veri analiz el kitabı [Data analysis handbook for social sciences]. Pegem Akademi.
18. Chen, F.F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464 504. https://doi.org/10.1080/10705510701301834
19. Cheung, G.W., & Rensvold, R.B. (2002). Evalutaing goodness-of-fit indexes for testing measurement invariance. Structual Equation Modelling, 9(2), 223-255. https://doi.org/10.1207/S15328007SEM0902_5
20. Comrey, A.L., & Lee, H.B. (1992). A first cource in factor analysis. Lawrence Erlbaum Associates, Inc.
21. Chalmers, R.P (2012). “mirt: A multidimensional item response theory package for the R environment.” Journal of Statistical Software, 48(6), 1 29. https://doi.org/10.18637/jss.v048.i06
22. Crocker, L., & Algina, J. (2008). Introduction to classical and modern test theory. Cengage Learning.
23. Darling, N. (2007). Ecological systems theory: The person in the center of the circles. Research in Human Development, 4(3), 203-217.
24. De Ayala, R.J. (2009). The theory and practice of item response theory. The Guilford Press.
25. Doğan, A. (2010). Ekolojik sistemler kuramı çerçevesinde akran zorbalığı incelemesi [Ecelogical systems model as a framework for bullying]. Turk J Child Adolesc Ment Health, 17(3), 149-162.
26. Duyan, V., Gelbal, S., & Var, E.Ç. (2013). Sosyal ilişki unsurları ölçeğinin Türkçeye uyarlama çalışması [The adaptation study of the provision of social relations scale to Turkish]. Hacettepe University Journal of Education, 44(44), 159-169.
27. Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Ersbaum.
28. Erkuş, A. (2016). Psikolojide ölçme ve ölçek geliştirme-1 [Education and development in psychology-1]. Pegem Akademi.
29. Espelage, D.L. (2014). Ecological theory: Preventing youth bullying, aggression, and victimization. Theory into Practice, 53(4), 257 264. https://doi.org/10.1080/00405841.2014.947216
30. Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurment, 58(3), 357-381. https://doi.org/10.1177/0013164498058003001
31. Ferhan, M. (2018). The psychometric characteristics of PISA 2012 mathematics interest scale by classical test theory and item response theory. [Master’s dissertation]. Hasan Kalyoncu University, Gaziantep]. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
32. Gençtanırım, D. (2015). Ergen intiharlarının önlenmesi: Ekolojik bakış açısı [Prevention Adolescents Suicide: Ecological Perspective]. Ahi Evran Üniversitesi Kırşehir Eğitim Fakültesi Dergisi (KEFAD), 16(1), 151-164.
33. Hair, J.F., Hult, G.T., Ringle, C.M. & Sarstedt M. (2014). A Primer on partial least squares structural equation modeling (PLS-SEM). SAGE Publications.
34. Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Sage Puplications.
35. Hambleton, R.K., & Jones, R.W. (1993). Comparison of classsical test theory and item response theory and their pplications to test development. Ecuational Measurement: Issues and Practice, 12(3), 38-47. https://doi.org/10.1111/j.1745-3992.1993.tb00543.x
36. Ho, R. (2006). Handbook of univariate and multivariate data analysis and interpretation with SPSS. Taylor & Francis Group.
37. Hong, J.S., & Eamon, M.K. (2012). Students’ perceptions of unsafe schools: An ecelogical systems analysis. Journal of Child and Family Studies, 21(428-438).
38. Karakılıç, M. (2009). An investigation of attitude scale measuring students attitudes toward physical education trough psychometric theories. [Doctoral dissertation] Ankara University, Ankara. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
39. Katsikatsou, M., Moustaki, I., Yang-Wallentin, F., & Joreskog, K. (2012). Pairwise likelihood estimation for factor analysis models with ordinal data. Computational Statistics and Data Analysis, 56(12), 4243- 4258. https://doi.org/10.1016/j.csda.2012.04.010
40. Kelecioğlu, H. (1992). Güdülenme [Motivation]. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 7, 175-181.
41. Kline, R.B. (2016). Principles and practice of structural equation modeling. The Guilford Press.
42. Koğar, H., & Yılmaz Koğar, E. (2015). Comparison of different estimation methods for categorical and ordinal data in confirmatory factor analysis. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 351 364 https://doi.org/10.21031/epod.94857
43. Kopan, D. (2019). Evaluation of nutrition habits the second grade students in Seferihisar region with the ecological framework [Master’s dissertation]. Ege University, İzmir. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
44. Köse, A. (2015). Aşamalı tepki modeli ve klasik test kuramı altında elde edilen test ve madde parametrelerinin karşılaştırılması [Comparison of test and item parameters under graded response model (IRT) and classical test theory]. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 15(2), 184-197. https://doi.org/10.17240/aibuefd.2015.15.2-5000161319
45. Leonard, J. (2011). Using Bronfenbrenner’s ecological theory to understand community partnerships: A historical case study of one urban high school. Urban Education, 0042085911400337. https://doi.org/10.1177/0042085911400337
46. Mapp, K.L. (2002). Having their say: Parents describe how and why they are involved in their children's education. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA, April, 1-5.
47. Meredith, W. (1993). Measurement invariance factor analysis and factorial invariance. Psychometrika, 58(4), 525-543.
48. Nartgün, Z. (2002). The investigation of item and scale properties of likert type scale and metric scale measuring the same attitude according to classisical test theory and item response theory [Doctoral dissertation]. Hacettepe University, Ankara. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
49. Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric Theory. McGraw-Hill.
50. Onwuegbuzie, A.J., Collins, K.M.T. & Frels, R.K. (2013). Foreword: Using Bronfenbrenners’s Ecological Systems Theory to frame quantitative, qualitative, and mixed reserch. International Journal of Multiple Research Approaches, 7(1), 2 8. https://doi.org/10.5172/mra.2013.7.1.2
51. Özbay, Y. (2018). Eğitim psikolojisi [Educational psychology]. Pegem Akademi.
52. Özenç, E.G., & Doğan, M.C. (2014). Ekolojik kurama dayalı işlevsel okuryazarlık yaşantısı ölçeğinin geliştirilmesi ve geçerlik güvenirlik çalışması [The development of the functional literacy experience scale based upon ecological theory (FLESBUET) and validity-reliability Study]. Educational Sciences: Theory & Practice, 14(6), 2239-2258. https://doi.org/10.12738/estp.2014.6.1791
53. R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
54. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
55. Santrock, J.W. (2011). Life-span development. The McGraw-Hill.
56. Sarı, H.İ., & Karaman, M.A. (2018). Gaining a better understanding of general mattering scale: An application of classical test theory and item response theory. İnternational Journal of Assessment Tools in Education, 5(4), 668-681. https://doi.org/10.21449/ijate.453337
57. Sarıer, Y. (2016). Türkiye’de öğrencilerin akademik bşarısını etkileyen faktörler: Bir meta-analiz çalışması [The factors that affects students' academic achievement in Turkey: A meta-analysis study]. Hacettepe University Journal of Education, 31(3), 609-627. https://doi.org/10.16986/HUJE.2016015868
58. Schumacker, R.E., & Lomax, R.G. (2004). A beginner’s guide to structural equation modelling. Lawrence Erlbaum Associates.
59. Sezgin, F., Koşar, D., & Koşar, S. (2016). Liselerde akademik başarısızlık: Nedenleri ve önlenmesine ilişkin öğretmen ve okul yöneticilerinin görüşleri [Teachers’ and school administrators’ views on reasons and prevention of academic failure in high schools]. İnönü University Journal of the Faculty of Education, 17(1), 95-111. https://doi.org/10.17679/iuefd.17119535
60. Shaffer, D.R. (2009). Social and personality development. Wadsworth.
61. Shelton, L.G. (2019). The Bronfenbrenner primer: A guide to develecology. Routledge.
62. Steenkamp, J., & Baumgartner, H. (1998). Assesing measurement invariance in cross national consumer research. Journal of Consumer Research, (25), 78 90. https://doi.org/10.1086/209528
63. Tabachnick, B.G., & Fidell, L.S. (2013). Using multivariate statistics. Pearson.
64. Tudge, J.R.H., Mokrova, İ., Hatfield, B.E., & Karnik, R.B. (2009). Uses and misuses of Bronfenbrenner’s bioecological theory of human development. Journal of Family Theory and Review, 1, 198– 210. https://doi.org/10.1111/j.1756-2589.2009.00026.x
65. Tuncer, M., & Bahadır, F. (2017). Ortaokul öğrenci görüşlerine göre başarısızlığın nedenleri [Reasons for underachievement by secondary schools’ students opinions]. Kahramanmaraş Sütçüimam Üniversitesi Eğitim Dergisi, 1(1), 1-11.
66. Turner, R.J., Frankel, B.G., & Levin, D.M. (1983). Social support: conceptualization, measurement, and implications for mental health. In J. R. Greeley (Ed.), Research in community and mental health (67-111). JAI Press.
67. Uysal, M. (2015). An investigation of psychometric properties of research self-efficacy scale according to classical test theory and item response theory [Master’s dissertation]. Gazi University, Ankara]. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
68. Vallerand, R.J., Blais, M.R., Briére, N.M., & Pelletier, L.G. (1989). Construction et validation de l’echelle de motivation en education (EME) [Construction and validation of the eçhelle de motivation en education (EME)]. Canadian Journal of Behavioral Sciences, 21, 323-349.
69. Vallerand, R.J., Pelletier, L.G., Blais, M.R., Briére, N.M., Senécal C., & Valliéres, E.F. (1992). The academic motivation scale: A measure of intrinsic, extrinsic, and amotivation in education. Educational and Psychological Measurement, 52(4), 1003-1017.
70. Watkins, M.W. (2000). Monte Carlo PCA for parallel analysis [Computer software]. Ed & Psych Associates.
71. Yaşar, M., & Aybek, E.C. (2019). Üniversite öğrencileri için bir yılmazlık ölçeğinin geliştirilmesi: Madde tepki kuramı temelinde geçerlilik ve güvenirlilik çalışması [A resilience scale development for university students: Validity and reliability study based on item response theory]. Elementary Education Online, 18(4), 1687-1699. 10.17051/ilkonline.2019.635031
72. Yaşar, M. (2019). Development of a “Perceived Stress Scale” based on classical test theory and graded response model. International Journal of Assessment Tools in Education, 6(3), 522-538. https://doi.org/10.21449/ijate.626053
73. Yurdugül, H. (2005). Ölçek geliştirme çalışmalarında kapsam geçerliği için kapsam geçerlik indekslerinin kullanılması [Using content validity indexes for content validity in scale development studies]. XIV. Ulusal Eğitim Bilimleri Kongresi, Pamukkale Üniversitesi Eğitim Fakültesi, 28-30 Eylül, Denizli.
74. Yurt, E., & Bozer, E.N. (2015). Akademik Motivasyon Ölçeğinin Türkçeye Uyarlanması [The adaptation of the academic motivation scale for turkish context]. Gaziantep University Journal of Social Sciences, 14(3), 669-685. https://doi.org/10.21547/jss.256759
75. Zorbaz, O., & Bilge, F. (2019). Suça sürüklenen çocukların değerlendirmesinde ekolojik sistem yaklaşımının kullanımı: Olgu sunumu [The use of the ecological system approach in assessment of juvenile delinquency: A case study]. Sosyal Politika Çalışmaları Dergisi, 19(44), 793-813. https://doi.org/10.21560/spcd.v19i49119.506405