Attitude-Achievement Paradox and Countries' Extreme-Response Style Tendenciess: The Case of PISA 2015

Abstract

Programme for International Student Assessment (PISA), which is one of the large-scale exams, evaluates countries' achievements in science, mathematics, and reading of 15-year-old students. In PISA, besides success tests, student, teacher, and school questionnaires are applied, and affective features related to success are measured. The aim of this study is to examine the extreme response style in countries where the attitude-success paradox is seen in PISA 2015, and to determine the relationship between the economic, social, and cultural status of the countries. For these purposes, the IR-tree method was used. In the research, a total of 7486 student data was used by selecting 10% sub-samples randomly from the samples of 14 countries where the attitude-success paradox was seen. In order to measure students' attitudes towards science, success motivation and enjoyment of science scales were used. In the study, the group variable showing the attitude-success paradox of the countries and the SES variables showing their socio-economic status are independent variables. In the analysis of the data, lme4 and irtrees packages were used under the R program. Three different models were established to examine the extreme response styles of the countries. It was observed that the model that measures one latent feature and two extreme response styles among the established models provides the best fit with the data. According to the results obtained from this model, it was seen that the extreme response style was effective in explaining the attitude-success paradox. In addition, it was concluded that the SES variable did not have a significant effect on the extreme response style.

Keywords

• IR-Tree
• Attitude
• Success
• Extreme Response Style

Tutum Başarı Paradoksu ve Ülkelerin Uç Tepki Stili Eğilimleri: PISA 2015 Örneği

Abstract

Geniş ölçekli sınavlardan biri olan Uluslararası Öğrenci Değerlendirme Programı ülkelerin 15 yaş grubu öğrencilerinin fen, matematik ve okuma alanındaki başarılarını değerlendirmektedir. Uluslararası Öğrenci Değerlendirme Programı’nda başarı testlerinin yanında öğrenci, öğretmen ve okul anketleri de uygulanarak başarı ile ilişkili duyuşsal nitelikler de ölçülmektedir. Çalışmanın amacı, Uluslararası Öğrenci Değerlendirme Programı 2015’te tutum-başarı paradoksunun görüldüğü ülkelerde uç tepki stilini incelemek ve uç tepki stili sergileme ile ülkelerin ekonomik, sosyal ve kültürel statüleri arasındaki ilişkiyi belirlemektir. Bu amaçlar doğrultusunda IR-tree yöntemi kullanılmıştır. Araştırmada tutum-başarı paradoksunun görüldüğü 14 ülkenin örnekleminden rastgele %10’luk alt örneklemler seçilerek toplam 7486 öğrenci verisi kullanılmıştır. Öğrencilerin fene yönelik tutumlarını ölçmek amacıyla başarı motivasyonu ve fenden hoşlanma ölçeklerinden yararlanılmıştır. Araştırmada ülkelerin tutum-başarı paradoksunu gösteren grup değişkeni ile sosyoekonomik statülerini gösteren SES değişkenleri bağımsız değişkenlerdir. Verilerin analizinde R programı altında yer alan lme4 ile irtrees paketleri kullanılmıştır. Ülkelerin uç tepki stillerini incelemek amacıyla üç farklı model kurulmuştur. Kurulan modeller arasında bir gizil özellik ve iki uç tepki stilini ölçen modelin veriye en iyi uyumu sağladığı görülmüştür. Bu modelden elde edilen sonuçlara göre, tutum-başarı paradoksunu açıklamada uç tepki eğiliminin etkili olduğu görülmüştür. Ayrıca SES değişkenin ise uç tepki eğiliminde anlamlı bir etkisinin olmadığı sonucu ortaya çıkmıştır.

Keywords

• IR-Tree
• Tutum
• Başarı
• Uç Tepki Stili

References

1. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390-412.
2. Bachman, J. G., & O’Malley, P. M. (1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48, 491-509.
3. Bandura, A. (1994). Self-efficacy. Encyclopedia of Human Behavior, 4, 71–81.
4. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48.
5. Birnbaum, A. L. (1968). Some latent trait models and their use in inferring an examinee's ability. Statistical Theories of Mental Test Scores.
6. Bolt, D. M., & Johnson, T. R. (2009). Addressing score bias and DIF due to individual differences in response style. Applied Psychological Measurement, 33(5), 335–352.
7. Bolt, D. M., & Newton, J. R. (2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71(5), 814-833.
8. Böckenholt, U. (2013). Modeling multiple response processes in judgment and choice. Decision, 1(S), 83–103.

9. Böckenholt, U. (2017). Measuring response styles in Likert items. Psychological Methods, 22(1), 69-83.
10. Böckenholt, U., & Meiser, T. (2017). Response style analysis with threshold and multi‐process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70(1), 159-181.
11. Buckley, J. (2009). Cross-national Response Styles in International Educational Assessment: Evidence from PISA 2006. NCES Conference on the Program for International Student Assessment: What we can learn from PISA. https://edsurvey.rti.org/PISA sayfasından erişilmiştir.
12. Büyüköztürk, Ş. (2013). Sosyal bilimler için veri analizi el kitabı. Ankara: Pegem.
13. Cronbach, L. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475–494.
14. De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48(1), 1-28.
15. De Boeck, P., & Wilson, M. (2004). Explanatory item response models: A generalized linear and nonlinear approach. Springer Science & Business Media.
16. DiTrapani, J., Jeon, M., De Boeck, P., & Partchev, I. (2016). Attempting to differentiate fast and slow intelligence: Using generalized item response trees to examine the role of speed on intelligence tests. Intelligence, 56, 82-92.
17. Doran, H., Bates, D., Bliese, P., & Dowling, M. (2007). Estimating the multilevel Rasch model: With the lme4 package. Journal of Statistical Software, 20(2), 1-18.
18. Eccles, J. S., Midgley, C., Wigfield, A., Miller, C., Reuman, D., Flanagan, C., & MacIver, D. (1993). Development during adolescence: The impact of stage-environment fit on young adolescents’ experiences in schools and in families. American Psychologist, 48(2), 90–101.
19. Eid, M., & Rauber, M. (2000). Detecting measurement invariance in organizational surveys. European Journal of Psychological Assessment, 16(1), 20-30.
20. Ersan, O., & Rodriguez, M. C. (2020). Socioeconomic status and beyond: a multilevel analysis of TIMSS mathematics achievement given student and school context in Turkey. Large-Scale Assessments in Education, 8(1), 1-32.
21. Falk, C. F., & Cai, L. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21(3), 328-347.
22. He, J., & Van de Vijver, F. (2016). Correcting for scale usage differences among Latin American countries, Portugal, and Spain in PISA. Electron. J. Educ. Res. Assess. Eval, 22(1), 1-12.
23. Heine, S. J., Kitayama, S., & Lehman, D. R. (2001). Cultural differences in self-evaluation: Japanese readily accept negative self-relevant information. Journal of Cross-Cultural Psychology, 32(4), 434-443.
24. Heine, S. J., Lehman, D. R., Peng, K., & Greenholtz, J. (2002). What's wrong with cross-cultural comparisons of subjective Likert scales? The reference-group effect. Journal of Personality and Social Psychology, 82(6), 90-918.
25. Heine, S. J., Takata, T., & Lehman, D. R. (2000). Beyond self-presentation: Evidence for self-criticism among Japanese. Personality and Social Psychology Bulletin, 26(1), 71–78.
26. İlgün-Dibek, M. (2018). TIMSS 2015 uygulamasında tutum-başarı paradoksunun uç tepki stiline göre çok boyutlu madde tepki kuramı ile modellenmesi. (Doktora tezi). https://tez.yok.gov.tr sayfasından erişilmiştir.
27. İlgün-Dibek, M. (2019). Examination of the extreme response style of students using IR-tree: The case of TIMSS 2015. International Journal, 6(2), 300-313.
28. İlgün-Dibek, M., Yavuz, H. Ç., & Çokluk-Bökeoğlu, Ö. (2018). Tutum - başarı paradoksunda tepki stillerinin rolü: Dokuz ülkenin karşılaştırılması. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 18(2), 932-952.
29. Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48(3), 1070-1085.
30. Jeon, M., & Rijmen, F. (2016). A modular approach for item response theory modeling with the R package flirt. Behavior Research Methods, 48(2), 742-755.
31. Jin, K. Y., & Wang, W. C. (2014). Generalized IRT models for extreme response style. Educational and Psychological Measurement, 74(1), 116-138.
32. Johnson, T. R., & Bolt, D. M. (2010). On the use of factor-analytic multinomial logit item response models to account for individual differences in response style. Journal of Educational and Behavioral Statistics, 35(1), 92–114.
33. Kankaraš, M., & Moors, G. (2011). Measurement equivalence and extreme response bias in the comparison of attitudes across Europe. Methodology 7(2), 68-80.
34. Karaağaç, Z. (2019). Ekonomik, sosyal ve kültürel statünün temel eğitimden ortaöğretime geçiş üzerine etkisi. (Doktora tezi). https://tez.yok.gov.tr sayfasından erişilmiştir.
35. Karasar, N. (2012). Bilimsel araştırma yöntemleri. Ankara: Nobel.
36. Kennedy, A., & Trong, K. (2006). A comparison of fourth-graders’ academic self-concept and attitudes toward reading, mathematics, and science in PIRLS and TIMSS countries. In 2nd IEA International Research Conference, Washington DC.
37. Kyllonen, P., Burrus, J., Roberts, R., & Van de Gaer, E. (2012). Cross-cultural comparative questionnaire issues. Paper prepared at the PISA.
38. LaHuis, D. M., Blackmore, C. E., Bryant-Lees, K. B., & Delgado, K. (2019). Applying item response trees to personality data in the selection context. Organizational Research Methods, 22(4), 1007-1018.
39. Lang, J. W., Lievens, F., De Fruyt, F., Zettler, I., & Tackett, J. L. (2019). Assessing meaningful within-person variability in likert-scale rated personality descriptions: An IRT tree approach. Psychological Assessment, 31(4), 474.
40. Lee, J. (2009). Universals and specifics of math self-concept, math self-efficacy, and math anxiety across 41 PISA 2003 participating countries. Learning and Individual Differences, 19(3), 355–365.
41. Leventhal, B. (2017). Extreme response style: Which model is best? (Doctoral’s thesis). ProQuest Dissertations and Theses database. (UMI No. 10645856).
42. Lie, S., & Turmo, A. (2007). Cross-country comparability of students self-reports–evidence from the PISA 2003 study. Nordic Studies in Education, 27(4), 343-354.
43. Liu, M., Lee, S., & Conrad, F. G. (2015). Comparing extreme response styles between agree-disagree and item-specific scales. Public Opinion Quarterly, 79(4), 952-975.
44. López-Sepulcre, A., De Bona, S., Valkonen, J. K., Umbers, K. D., & Mappes, J. (2015). Item response trees: A recommended method for analyzing categorical data in behavioral studies. Behavioral Ecology, 26(5), 1268-1273.
45. Lu, Y. (2012). A multilevel multidimensional item response theory model to address the role of response style on measurement of attitudes in PISA 2006. (Doctoral’s thesis). ProQuest Dissertations and Theses database. (UMI No. 3517105).
46. Lu, Y., & Bolt, D. M. (2015). Examining the attitude-achievement paradox in PISA using a multilevel multidimensional IRT model for extreme response style. Large-Scale Assessments in Education, 3(1), 1-18.
47. Maaz, K., Trautwein, U., Lüdtke, O., & Baumert, J. (2008). Educational transitions and differential learning environments: How explicit between‐school tracking contributes to social inequality in educational outcomes. Child Development Perspectives, 2(2), 99-106.
48. Marsh, H. W., & Hau, K. T. (2004). Explaining paradoxical relations between academic self-concepts and achievements: Cross-cultural generalizability of the internal/external frame of reference predictions across 26 countries. Journal of Educational Psychology, 96(1), 56-67.
49. Marsh, H. W., Trautwein, U., Lüdkte, O., Köller, O., & Baumert, J. (2005). Academic self concept, interest, and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397–416.
50. Marsh, H. W., Seaton, M., Trautwein, U., Lüdtke, O., Hau, K. T., O’Mara, A. J., & Craven, R. G. (2008). The big-fish–little-pond-effect stands up to critical scrutiny: Implications for theory, methodology, and future research. Educational Psychology Review, 20(3), 319-350.
51. Maydeu-Olivares, A., & Coffman, D. L. (2006). Random intercept item factor analysis. Psychological Methods, 11(4), 344–362.
52. Meiser, T. & Machunsky, M. (2008). The personal structure of personal need for structure. European Journal of Psychological Assessment, 24(1), 27-34.
53. Milli Eğitim Bakanlığı. (MEB). (2016). PISA 2015 Ulusal Ön Raporu. Ankara: MEB.
54. Milli Eğitim Bakanlığı. (2010). PISA 2009 Ulusal Ön Rapor. Ankara: Eğitim Araştırma ve Geliştirme Dairesi Başkanlığı.
55. Min, I., Cortina, K. S., & Miller, K. F. (2016). Modesty bias and the attitude-achievement paradox across nations: A reanalysis of TIMSS. Learning and Individual Differences, 51, 359-366.
56. Moors, G. (2003). Diagnosing response style behavior by means of a latent-class factor approach. Socio-demographic correlates of gender role attitudes and perceptions of ethnic discrimination reexamined. Quality and Quantity, 37(3), 277-302.
57. Morren, M., Gelissen, J. P., & Vermunt, J. K. (2011). Dealing with extreme response style in cross-cultural research: A restricted latent class factor analysis approach. Sociological Methodology, 41(1), 13-47.
58. OECD (2014). PISA 2012 Technical Report. https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf sayfasından erişilmiştir.
59. OECD (2016). PISA 2015 results (Volume I): Excellence and equity in education. Paris: OECD Publishing.
60. Okumura, T. (2014). Empirical differences in omission tendency and reading ability in PISA: an application of tree-based item response models. Educational and Psychological Measurement, 74(4), 611-626.
61. Oral, I., & McGivney, E. J. (2014). Türkiye’de eğitim sisteminde eşitlik ve akademik başarı: Araştırma raporu ve analiz. İstanbul: Sabancı Üniversitesi Yayınları.
62. Özgürlük, B., Ozarkan, H. B., Arıcı, Ö., & Taş, U. E. (2016). PISA 2015 Türkiye Ulusal Raporu [PISA 2015 Turkey national report]. Ölçme, Değerlendirme ve Sınav Hizmetleri Genel Müdürlüğü. https://pisa.meb.gov.tr/wp-content/uploads/2014/11/PISA2 015_Ulusal Rapor.pdf sayfasından erişilmiştir.
63. Özkan, U. B. (2020). Öğrencilerde eudaimonianın ve akademik başarının yordayıcısı olarak ekonomik, sosyal ve kültürel düzey. Yaşadıkça Eğitim, 34(2), 344-359.
64. Park, M., & Wu, A. D. (2019). Item response tree models to investigate acquiescence and extreme response styles in likert-type rating scales. Educational and Psychological Measurement, 79(5), 911-930.
65. Plieninger, H. (2017). Mountain or molehill? A simulation study on the impact of response styles. Educational and Psychological Measurement, 77(1), 32-53.
66. Plieninger, H., & Meiser, T. (2014). Validity of multi process IRT models for separating content and response styles. Educational and Psychological Measurement, 74(5), 875-899.
67. Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185–205.
68. Shen, C., & Tam, H. K. (2008). The paradoxical relationship between student achievement and self-perception: A corss-national analysis based on three waves of TIMSS data. Educational Research and Evaluation, 14(1), 87–100.
69. Thissen-Roe, A., & Thissen, D. (2013). A two-decision model for responses to Likert-type items. Journal of Educational and Behavioral Statistics, 38(5), 522-547.
70. Tutz, G., & Berger, M. (2016). Response styles in rating scales: Simultaneous modeling of content-related effects and the tendency to middle or extreme categories. Journal of Educational and Behavioral Statistics, 41(3), 239-268.
71. Üstün, U., Özdemir, E., Cansız, M., & Cansız, N. (2020). Türkiye’deki öğrencilerin fen okuryazarlığını etkileyen faktörler nelerdir? PISA 2015 verisine dayalı bir hiyerarşik doğrusal modelleme çalışması. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 35(3), 720-732.
72. Van de Gaer, E., & Adams, R. (2010). The modeling of response style bias: An answer to the attitude-achievement paradox? The Annual Conference of the American Educational Research Association’da sunulmuş bildiri, Denver, Colorado, USA.
73. Van de Gaer, E., Grisay, A., Schulz, W., & Gebhardt, E. (2012). The reference group effect: An explanation of the paradoxical relationship between academic achievement and self-confidence across countries. Journal of Cross-Cultural Psychology, 43(8), 1205–1228.
74. Von Davier, M. (2005). MDLTM: Software for the general diagnostic model and for estimating mixtures of multidimensional discrete latent traits models [Computer software]. Princeton, NJ: ETS.
75. Wetzel, E., & Carstensen, C. H. (2017). Multidimensional modeling of traits and response styles. European Journal of Psychological Assessment, 33(5), 352–364.
76. Wetzel, E., Carstensen, C. H., & Böhnke, J. R. (2013). Consistency of extreme response style and non-extreme response style across traits. Journal of Research in Personality, 47(2), 178-189.
77. Yıldırım, A., Özgürlük, B., Parlak, B., Gönen, E., & Polat, M. (2016). TIMSS 2015 Ulusal Matematik ve Fen Bilimleri Ön Raporu: 4. ve 8. Sınıflar. Ankara: T.C. Milli Eğitim Bakanlığı Ölçme, Değerlendirme ve Sınav Hizmetleri Genel Müdürlüğü.
78. Yolsal, H. (2016). Öğrencilerin sosyoekonomik ve kültürel statülerinin PISA 2012 başarıları üzerindeki etkisinin incelenmesi. Sosyal Bilimler Araştırma Dergisi, 5(3), 7-27.