Examination of the Extreme Response Style of Students using IRTree: The Case of TIMSS 2015

Öz

In the literature, response style is one of the factors causing an achievement-attitude paradox and threatens the validity of the results obtained from studies. In this regard, the aim of this study is two-fold. Firstly, it attempts to determine which item response tree (IRTree) models based on the generalized linear mixed model (GLMM) approach (random intercept, random intercept with the fixed effect of extreme response and random intercept-slope model) best fit the Trends in International Mathematics and Science Study (TIMSS) 2015 data. Secondly, it purports to explore how the extreme response style affects students’ attitudes toward mathematics of students. This study is both basic research and descriptive research in terms of seeking for answers for two different research questions. For the sample of this research, 15 countries were randomly selected among countries participated in TIMSS 2015. The students’ responses to items measuring attitude in the student questionnaire were analyzed with the packages “lme4” and “irtrees” in R software. When the model fit indices were evaluated, the random intercept-slope model was found to be the best fit to the data. According to this model, the extreme response style explains a significant amount of variances in the students’ attitude toward mathematics. Additionally, students with a negative attitude toward mathematics were found to have an extreme response style. It was concluded that an extreme response style had an effect on students’ attitude.

Anahtar Kelimeler

• Attitude
• Extreme Response Style
• Item Response Tree
• TIMSS

Examination of the Extreme Response Style of Students using IRTree: The Case of TIMSS 2015

Abstract

In the literature, response style is one of the factors causing an achievement-attitude paradox and threatens the validity of the results obtained from studies. In this regard, the aim of this study is two-fold. Firstly, it attempts to determine which item response tree (IRTree) models based on the generalized linear mixed model (GLMM) approach (random intercept, random intercept with the fixed effect of extreme response and random intercept-slope model) best fit the Trends in International Mathematics and Science Study (TIMSS) 2015 data. Secondly, it purports to explore how the extreme response style affects students’ attitudes toward mathematics of students. This study is both basic research and descriptive research in terms of seeking for answers for two different research questions. For the sample of this research, 15 countries were randomly selected among countries participated in TIMSS 2015. The students’ responses to items measuring attitude in the student questionnaire were analyzed with the packages “lme4” and “irtrees” in R software. When the model fit indices were evaluated, the random intercept-slope model was found to be the best fit to the data. According to this model, the extreme response style explains a significant amount of variances in the students’ attitude toward mathematics. Additionally, students with a negative attitude toward mathematics were found to have an extreme response style. It was concluded that an extreme response style had an effect on students’ attitude.

Keywords

• Attitude
• Extreme Response Style
• Item Response Tree
• TIMSS

References

1. Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561–573.
2. Atkinson, J. W. (1957). Motivational determinants of risk-taking behavior. Psychological Review, 64, 359-373.
3. Bachman, J. G., O’Malley, P. M., & Freedman-Doan, P. (2010). Response styles revisited: Racial/ethnic and gender differences in extreme responding (Monitoring the Future Occasional Paper No. 72). Ann Arbor, MI: Institute for Social Research.
4. Bates, D., Maechler, M., Bolker, B. & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. doi:10.18637/jss.v067.i01
5. Bofah, E. A. and Hannula, M. S. (2015). TIMSS data in an African comparative perspective: Investigating the factors influencing achievement in mathematics and their psychometric properties. Large-Scale Assessments in Education, 3(1), doi:1.1186/s40536-015-0014-y
6. Bolt, D. M., & Newton, J. (2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71, 814-833.
7. Bolt, D., Wollack, J., & Suh, Y. (2012). Application of a multidimensional nested logit model to multiple-choice test items. Psychometrika, 77, 339–357.
8. Böckenholt, U. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665-678.

9. Böckenholt, U. (2017). Measuring response styles in Likert items. Psychological Methods, 22(1), 69–83. doi:10.1037/met0000106
10. Böckenholt, U. & Meiser, T. (2017). Response style analysis with threshold and multi-process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70, 159–181. doi:10.1111/bmsp.12086
11. Buckley, J. (2009, June). Cross-national response styles in international educational assessment: Evidence from PISA 2006. NCES Conference on the Program for International Student Assessment: What we can learn from PISA, Washington, DC.
12. Büyüköztürk, Ş. (2005). Sosyal Bilimler için veri analizi el kitabı[Data analysis handbook for social sciences]. 5. baskı. Pagem A Yayıncılık.
13. Bybee, R.,& McCrae, B. (2007). Scientific literacy and student attitudes: Perspectives from PISA 2006 science. International Journal of Science Education, 33, 7-26.
14. Cronbach, L. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475-494.
15. Culpepper, S. (2014). If at first you don’t succeed, try, try again: Applications of sequential IRT models to cognitive assessments. Applied Psychological Measurement, 38, 632– 644.
16. De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48, 1–28.
17. De Boeck P & Wilson M (2004). Explanatory item response models: A generalized linear and nonlinear approach. Springer-Verlag, New York.
18. Harter, S. (1999). The construction of the self: A developmental perspective. New York: Guildford Press.
19. Heide, M. & Gronhaug, K. (1992) The impact of response styles in surveys: a simulation study. Journal of the Market Research Society, 34, 215-231.
20. Hofstede, G. H. (2001). Cultures consequences: Comparing values, behaviors, institutions, and organizations across nations (2nd ed.). Thousand Oaks, California: Sage Publications, Inc.
21. Hooper, M, Mullis. I. V. S., & Martin, M.O. (2013). TIMSS 2015 Context Questionnaire Framework. Mullis, I.V.S. and Martin, M.O. (Eds.) TIMSS 2015 Assessment Frameworks. Retrieved January 15, 2019, from Boston College, TIMSS and PIRLS International Study Center website: http://timssandpirls.bc.edu/timss2015/frameworks.html
22. Hox J. 2002. Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum
23. Huang H-Y. (2016) Mixture random-effect IRT models for controlling extreme response style on rating scales. Frontiers Psychology, 7(1706), 1-15. doi: 10.3389/fpsyg.2016.01706
24. Ilgun Dibek, M., Bulut, O., Sahin Kursad, M., & Yavuz, H. C. (2018, July). Should students with disabilities have multiple opportunities in answering items? Paper presented at the International Testing Commission Conference, Montreal, QC, Canada
25. Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior research methods, 48(3), 1070–1085. doi: 10.3758/s13428-015-0631-y
26. Johnson, T.R. (2007). Discrete choice models for ordinal response variables: A generalization of the stereotype model. Psychometrika, 72, 489–504.
27. Johnson, R.B. and Christensen, L.B. (2008) Educational Research: Quantitative, Qualitative, and Mixed Approaches. 3rd Edition, Sage Publications, Inc., Los Angeles.
28. Kadijevich, D. (2008). TIMSS 2003: Relating dimensions of mathematics attitude to mathematics achievement. Zbornik instituta za Pedagogical Research, 40(2), 327–346. doi: 1.2298/ZIPI0802327K
29. Kidd, C. V. (1959). Basic research: Description versus definition. Science, 129, 368-371.
30. LaRoche, S. & Foy, P. (2016). Sample Implementation in TIMSS 2015. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and Procedures in TIMSS 2015 (pp. 5.1-5.175). Retrieved January 8, 2019, from Boston College, TIMSS & PIRLS International Study Center website: http://timss.bc.edu/publications/timss/2015-methods/chapter-5.html
31. LaRoche, S., Joncas, M., and Foy, P. (2016). Sample Design in TIMSS 2015. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and Procedures in TIMSS 2015 (pp. 3.1-3.37). Retrieved January 10, 2019, from Boston College, TIMSS & PIRLS International Study Center website: http://timss.bc.edu/publications/timss/2015-methods/chapter-3.html
32. Leventhal, B.C & Stone, C.A (2018). Bayesian analysis of multidimensional item response theory models: A discussion and illustration of three response style models, Measurement: Interdisciplinary Research and Perspective, 16(2), 114-128, doi: 10.1080/15366367.2018.1437306
33. Liu, M. (2015). Response Style and Rating Scales: The Effects of Data Collection Mode, Scale Format, and Acculturation (Unpublished doctoral dissertation). The University of Michigan.
34. Marsh, H. W., Trautwein, U., Lüdtke, O., Köller, O & Baumert, J. (2005). Academic self-concept, interest, grades and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397-416.
35. Masters, G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.
36. Mooi, E., Sarstedt, M., & Mooi-Reci, I. (2018). Market research: The process, data, and methods using Stata. Singapore: Springer.
37. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.
38. Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2016). TIMSS 2015 International Results in Mathematics.Retrieved January 10, 2019, from Boston College, TIMSS & PIRLS International Study Center website: http://timssandpirls.bc.edu/timss2015/international-results/
39. Nakagawa, S., and H. Schielzeth. 2013. A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4(2): 133-142. doi: 10.1111/j.2041-210x.2012.00261.x
40. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill
41. Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver & L. S. Wrightman (Eds.), Measures of Personality and Social Psychological Attitudes (Vol. 1). San Diego, CA: Academic Press.
42. Peterson, R.A, Rhi-Perez, P. & Albaum, G. (2012). A cross-national comparison of extreme response style measures. International Journal of Market Research, 56(1), 89-110.
43. Thissen-Roe, A., & Thissen, D. (2013). A two-decision model for responses to Likert-type Items. Journal of Educational and Behavioral Statistics, 38(5), 522-547.
44. Tutz, G. (1990). Sequential item response models with an ordered response. British Journal of Mathematical and Statistical Psychology, 43, 39–55.
45. Van de Gaer, E. & Adams, R. (2010, May). The Modeling of Response Style Bias: An Answer to the Attitude-Achievement Paradox?, paper presented at the annual conference of the American Educational Research Association, Denver, Colorado, USA.
46. Van de gaer, E., Grisay, A., Schulz, W. & Gebhardt, E. (2012). The reference group effect an explanation of the paradoxical relationship between academic achievement and self-confidence across countries. Journal of Cross-Cultural Psychology, 43(8), 1205-1228
47. Yavuz, H. C., Bulut, O., Ilgun Dibek, M., & Sahin Kursad, M. (2018, July). Providing revision opportunities in alternate assessments: An application of sequential IRT. Paper presented at the International Testing Commission Conference, Montreal, QC, Canada.