Investigating the sources of differential item functioning: A sample critical thinking motivation scale

Öz

Investigating the existence of items with differential item functioning (DIF) may provide more accurate comparisons of group differences in studies that aim to compare scores obtained in a test by groups with different characteristics. In the present study, a scale measuring critical thinking motivation that was adapted to the Turkish culture was applied to 817 participants, who were high school graduates, university students, and university graduates. The aim of the study was to examine whether the data collected from these participants had DIF or not. Hence, DIF analysis of the collected data was performed via the "lordif" function in the R "lordif" package. DIF was found to occur in twelve items, three of which were related to gender and nine to level of education. While it was revealed that the content of the items was the source of gender related DIF, the source of DIF related to level of education was found to be the language and expression of the items.

Anahtar Kelimeler

• Differential item functioning
• Critical thinking
• Critical thinking motivation

References

1. Altıntaş, Ö., & Kutlu, Ö. (2019). Investigating differential item functioning of Ankara University examination for foreign students by recursive partitioning analysis in the Rasch model. International Journal of Assessment Tools in Education, 6(4), 602–616. https://doi.org/10.21449/ijate.554212
2. Ateşok Deveci, N. (2008). Examination of Inter-university Board foreign language test in the frame of item bias. [Doctoral dissertation, Ankara University]. https://tez.yok.gov.tr
3. Athman Ernst, J., & Monroe, M. (2004). The effects of environment‐based education on students' critical thinking skills and disposition toward critical thinking. Environmental Education Research, 10(4), 507-522. https://doi.org/10.1080/1350462042000291038
4. Bar-Tal, D. (1978). Attributional analysis of achievement-related behavior. Review of Educational Research, 48(2), 259–271. https://doi.org/10.3102/00346543048002259
5. Baron, J. (1985). Rationality and intelligence. Cambridge University Press.
6. Büyüköztürk, Ş. (2021). Sosyal bilimler için veri analizi el kitabı [Data analysis handbook for social sciences]. PegemA.
7. Byrne, B.M., Oakland, T., Leong, F.T.L., Van De Vijver, F.J.R., Hambleton, R.K., Cheung, F. M., & Bartram, D. (2009). A critical analysis of cross-cultural research and testing practices: Implications for improved education and training in psychology. Training and Education in Professional Psychology, 3, 94-105. https://doi.org/10.1037/a0014516
8. Choi, S.W., Gibbons, L.E., & Crane, P.K. (2011). Lordif: An R package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations. Journal of Statistical Software, 39(8), 1. https://doi.org/10.18637/jss.v039.i08

9. Cole, D.A., Maxwell, S.E., Avery, R., & Salas, E. (1993). Multivariate group comparisons of variable systems: MANOVA and structural equation modeling. Psychological Bulletin, 114, 174-184. https://doi.org/10.1037/0033-2909.114.1.174
10. Crandall, V.C., Katkovsky, W., & Crandall, V.J. (1965). Children’s belief in their own control of reinforcement in intellectual-academic achievement situations. Child Development, 36, 91–109. https://doi.org/10.2307/1126783
11. Crane, P.K., Gibbons, L.E., Jolley, L., & Van Belle, G. (2006). Differential item functioning analysis with ordinal logistic regression techniques: DIF detect and difwithpar. Medical Care, 44(11), 115-123. https://www.jstor.org/stable/41219511
12. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Thomson Learning.
13. Davidov, E., Meuleman, B., Cieciuch, J., Schmidt, P., & Billiet, J. (2014). Measurement equivalence in cross-national research. Annual Review of Sociology, 40, 55-75. https://doi.org/10.1146/annurev-soc-071913-043137
14. Dewey, J. (1930). Human nature and conduct. The Modern Library.
15. do Nascimento, C.D., Peter, W.F., Ribeiro, I.M., Moreira, B.S., Lima, V.P., Kirkwood, R.N., & Bastone, A.C. (2021). Cross-cultural validity of the animated activity questionnaire for patients with hip and knee osteoarthritis: A comparison between the Netherlands and Brazil, Brazilian Journal of Physical Therapy, 000(xxxx), 1-8. http://doi.org/10.1016/j.bjpt.2021.06.002
16. Dönmez, B., & Kaya, F. (2016). Eleştirel Düşünme Motivasyonu Ölçeği’nin Türkçe’ye uyarlanması [Turkish adaptation study of Critical Thinking Motivational Scale]. HAYEF Journal of Education, 13 2(25), 159 173. https://dergipark.org.tr/tr/pub/iuhayefd/issue/24491/259590
17. Eccles, J.S., Adler, T.F., Futterman, R., Goff, S.B., Kaczala, C.M., & Meece, J.L. (1983). Expectancies, values and academic behaviors. In J. T. Spence (Ed.), Achievement and Achievement Motives (pp. 75–146). San Francisco Freeman.
18. Ennis, R.H. (1991). Critical thinking: A streamlined conception. Teaching Philosophy, 14, 5- 25. http://doi.org/10.1057/9781137378057_2
19. Ennis, R.H. (1993). Critical thinking assessment. Theory into Practice, 32, 179-186. https://doi.org/10.1080/00405849309543594
20. Ennis, R.H. (1996). Critical thinking dispositions: Their nature and assessability. Informal Logic, 18(2), 165-182. https://doi.org/10.22329/il.v18i2.2378
21. Ersözlü, A. N., & Arslan, M. (2009). The effect of developing reflective thinking on metacognitional awareness at primary education level in Turkey. Reflective Practice, 10, 683–695. https://doi.org/10.1080/14623940903290752
22. Facione, P.A., & Facione, N.C. (1992). The California critical thinking dispositions inventory. California Academic Press.
23. Farmer, H.S., & Vispoel, W.P. (1990). Attributions of female and male adolescents for real-life failure experiences. Journal of Experimental Education, 58(2), 127–140. https://doi.org/10.1080/00220973.1990.10806529
24. Feingold, A. (1994). Gender differences in personality: A meta analysis. Psychological Bulletin, 116, 429-456. https://doi.org/10.1037/0033-2909.116.3.429
25. Ferne, T., & Rupp, A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4, 113-148. https://doi.org/10.1080/15434300701375923
26. French, B.F., Hand, B., Nam, J., Yen, H.J., & Vazquez, J.A.V. (2014). Detection of differential item functioning in the Cornell Critical Thinking Test across Korean and North American students. Psychological Test and Assessment Modeling, 56(3), 275.
27. French, B.F., Hand, B., Therrien, W.J., & Vazquez, J.A.V. (2012). Detection of sex differential item functioning in the Cornell Critical Thinking Test. Europen Journal of Psychological Assessment, 28(3), 201-207. http://doi.org/10.1027/1015-5759/a000127
28. Frieze, I.H. (1975). Women’s expectations for and causal attributions of success and failure. In T. Mednick, S. Tangi, & L.W. Hoffman (Eds.), Women and achievement. Social and motivational analysis. (pp. 158–171). John Wiley and Sons.
29. Galic, Z., Scherer, K.T., & Leberton, J.M. (2014). Examining the measurement equivalence of the conditional reasoning test for aggression across U.S. and Croatian samples. Psychological Test and Assessment Modeling, 56, 195 216. http://darhiv.ffzg.unizg.hr/id/eprint/5547
30. Gallos, J.V. (1995). Gender and silence. Collage Teaching, 43(3), 101-105. http://doi.org/10.1080/87567555.1995.9925525
31. Garcia, J.M., Gallagher, M.W., O’Bryant, S.E., & Medina, L.D. (2021). Differential item functioning of the Beck Anxiety Inventory in a rural, multi-ethnic cohort. Journal of Affective Disorders, 293, 36-42. http://doi.org/10.1016/j.jad.2021.06.005
32. Garcia, T., & Pintrich, P.R. (1992). The effect of PBL curriculum on students’ motivation and self-regulation. Paper presented at The Biennial Conference of The European Association for Research on Learning and Instruction, Italy.
33. Gierl, M., Khaliq, S.N., & Boughton, K. (1999). Gender differential item functioning in mathematics and science: Prevalence and policy implications. In Improving Large-Scale Assessment in Education Symposium at the Annual Meeting of the Canadian Society for the Study of Education, Canada.
34. Gilligan, C. (1979). Woman’s place in man’s life cycle. Harvard Educational Review, 49, 431-446. https://doi.org/10.17763/haer.49.4.h136573541l3g463
35. Glevey, K.E. (2006). Promoting thinking skills in education. London Review of Education, 4(3), 291-302. http://doi.org/10.1080/14748460601044005
36. Glover, T.A., & Albers, C.A. (2007). Consideraitons for evaluating universal screening assessments. Journal of School Psychology, 45(2), 117 135. http://doi.org/10.1016/j.jsp.2006.05.005
37. Gök, B., Kabasakal, K.A., & Kelecioğlu, H. (2014). PISA 2009 öğrenci anketi tutum maddelerinin kültüre göre değişen madde fonksiyonu açısından incelenmesi [Analysis of attitude items in PISA 2009 student questionnaire in terms of differential item functioning based on culture]. Journal of Measurement and Evaluation in Education and Psychology, 5(1), 72-87. https://doi.org/10.21031/epod.64124
38. Halpern, D.F. (1998). Teaching critical thinking for transfer across domains. American Psychologist, 53, 449-455. https://doi.org/10.1037/0003-066X.53.4.449
39. Hambelton, R., & Rogers, J. (1995). Item bias review. Practical Assessment, Research, and Evaluation, 4(6), https://doi.org/10.7275/jymp-md73
40. Hambleton, R.K., & Rovinelli, R.J. (1986). Assessing the dimensionality of a set of test items. Applied Psychological Measurement, 10(3), 287 302. https://doi.org/10.1177/014662168601000307
41. Hambleton, R.K., & Swaminathan, H. (1989). Item response theory: Principles and applications. Kluwer Nijhoff.
42. Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Sage.
43. Ingle, C.O. (2007). Predictors of critical thinking ability among college students. [Doctoral dissertation]. Available from ProQuest Dissertations and Theses Database. (UMI No. 3263681).
44. Jensen, A.R. (1980). Bias in mental testing. Free Press.
45. Kalaycioğlu, D.B., & Kelecioğlu, H. (2011). Öğrenci Seçme Sınavı’nın madde yanlılığı açısından incelenmesi [Item Bias Analysis of the University Entrance Examination]. Education and Science, 36(161), 3-11.
46. Karakaya, İ., & Kutlu, Ö. (2012). Seviye Belirleme Sınavındaki Türkçe alt testlerinin madde yanlılığının incelenmesi [An Investigation of Item Bias in Turkish Sub Tests in Level Determination Exam]. Education and Science, 37(165). http://egitimvebilim.ted.org.tr/index.php/EB/article/view/1342
47. King, P.M., Wood, P.K., & Mines, R.A. (1990). Critical thinking among college and graduate students. The Review of Higher Education, 13, 167 186. http://doi.org/10.1353/rhe.1990.0026
48. Kholberg, L. (1973). Continuities and discontinuities in childhood and adult moral development revisited. In P. B. Baltes & L. R. Goutlet (Eds.), Lifespan developmental psychology: Research and theory (pp. 179-204). Academic Press.
49. Kloosterman, P. (2001). Attributions, performance following failure, and motivation in mathematics. In E. Fennema & G.C. Leder (Eds.), Mathematics and gender. Teachers College Press.
50. Köse, İ.A. (2015). PISA 2009 öğrenci anketi alt ölçeklerinde (Q32-Q33) bulunan maddelerin değişen madde fonksiyonu açısından incelenmesi [Investigation of items in PISA 2009 student questionnaire subscales (Q32-Q33) in terms of differential item functioning]. Kastamonu Education Journal, 23(1), 227 240. https://dergipark.org.tr/en/pub/kefdergi/issue/22600/241461
51. Kurnaz, F.B., & Kelecioğlu, H. (2008). Investigation of Peabody Picture Vocabulary Test from the point of item bias. World Applied Sciences Journal, 3(2), 231-239. https://www.academia.edu/7282678/Investigation_of_Peabody_Picture_Vocabulary_Test_from_the_point_of_item_bias_peabody_picture_vocabulary_test
52. Kurnaz Adıbatmaz, F.B., & Yıldız, H. (2020). The Effects of distractors to differential item functioning in Peabody Picture Vocabulary Test. Journal of Theoretical Educational Science, 13(3), 530-547. https://dergipark.org.tr/tr/pub/akukeg/issue/54987/621581
53. Lau, C., Chiesi, F., Saklofske, D.H., Yan, G., & Li, C. (2020). How essential is the essential resilience scale? Differential item functioning of Chinese and English versions and criterion validity. Personality and Individual Differences, 155, 109666. http://doi.org/10.1016/j.paid.2019.109666
54. Lau, C., Swindall, T., Chiesi, F., Quilty, L.C., Chen, H.C., Chan, Y.C., ... & Torres-Marín, J. (2023). Cultural differences in how people deal with ridicule and laughter: Differential item functioning between the Taiwanese Chinese and Canadian English versions of the PhoPhiKat-45. European Journal of Investigation in Health, Psychology and Education, 13(2), 238-258. https://doi.org/10.3390/ejihpe13020019
55. Maller, S.J. (2001). Differential item functioning in the WISC-III: Item parameters for boys and girls in the national standardization sample. Educational and Psychological Measurement, 61(5), 793-817. https://doi.org/10.1177/00131640121971527
56. McLean, C.P., & Miller, N.A. (2010). Changes in critical thinking skills following a course on science and pseudoscience: A quasi-experimental study. Teaching of Psychology, 37, 85-90. https://doi.org/10.1080/00986281003626714
57. Mcpeck, J.E. (1990). Teaching critical thinking. Routledge.
58. Meece, J.L., Glienke, B. B., & Burg, S. (2006). Gender and motivation. Journal of School Psychology, 44(5), 351-373. https://doi.org/10.1016/j.jsp.2006.04.004
59. Mertler, C.A., Vannatta, R.A., & LaVenia, K.N. (2005). Advanced and multivariate statistical methods: Practical application and interpretation. Pyrczak. https://doi.org/10.4324/9781003047223
60. Millsap, R.E. (2012). Statistical approaches to measurement invariance. Routledge.
61. Muraki, E. (1992). A generalized partial credıt model: Applicatıon of an em algorithm. ETS Research Report Series. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
62. Nielsen, T., & Dammeyer, J. (2019). Measuring higher education students’ perceived stress: An IRT-based construct validity study of the PSS-10. Studies in Educational Evaluation, 63, 17-25. http://doi.org/10.1016/j.stueduc.2019.06.007
63. Osterlind, S.J. (1983). Test item bias. Sage.
64. Parsons, J., Adler, T.F., & Kaczala, C.M. (1984). Socialization of achievement attitudes and beliefs: Parental influences. Child Development, 53, 322 339. https://doi.org/10.2307/1128973
65. Paul, R.W. (1990). Critical thinking: What every person needs to survive in a rapidly changing world. Center for Critical Thinking and Moral Critique, Sonoma State University.
66. Perkins, D.N., Jay, E., & Tishman, S. (1993). Beyond abilities: A dispositional theory of thinking. Merrill-Palmer Quarterly, 39(1), 1-21. https://www.jstor.org/stable/23087298
67. Poortinga, Y.H. (1989). Equivalence of cross-cultural data: An overview of basic issues. International Journal of Psychology, 24(6), 737 756. https://doi.org/10.1080/00207598908247842
68. Ruble, D., Greulich, F., Pomerantz, E.M., & Gochberg, B. (1993). The role of gender-related processes in the development of sex differences in self-evaluation and depression. Journal of Affective Disorders, 29, 97–128. https://doi.org/10.1016/0165-0327(93)90027-H
69. Serin, Q., Serin, N.B., Saracaloğlu, A.S., & Ceylan, A. (2010). The examination of critical thinking styles of university students (TRNC Sample). Procedia Social and Behavioral Sciences, 9, 864–868. https://doi.org/10.1016/j.sbspro.2010.12.250
70. Steinberg, L., & Thissen, D. (2006). Using effect sizes for research reporting: examples using item response theory to analyze differential item functioning. Psychological Methods, 11, 402-415. https://doi.org/10.1037/1082-989X.11.4.402
71. Stump, T.E., Monahan, P., & Mchorney, C.A. (2005). Differential item functioning in the short portable mental status questionnaire. Research on Aging, 27(3), 355-384. https://doi.org/10.1177/0164027504273784
72. Schwartz, S., & Meyer, I.H. (2010). Mental health disparities research: The impact of within and between group analyses on tests of social stress hypotheses. Social Science & Medicine, 70(8), 1111-1118. https://doi.org/10.1016/j.socscimed.2009.11.032
73. Şengül Avşar, A., & Emons, W.H.M. (2021). A cross-cultural comparison of non-cognitive outputs towards science between Turkish and Dutch students taking into account detected person misfit. Studies in Educational Evaluation, 70(101053). http://doi.org/10.1016/j.stueduc.2021.101053
74. Tabachnick, B.G., & Fidell, L.S. (2007). Using multivariate statistics. Pearson.
75. Tedesco, J. (1991). Women’s ways of knowing/women’s ways of composing. Rhetoric Review, 9(2), 246-256. http://doi.org/10.1080/07350199109388931
76. Tsui, L. (2000). Effects of campus culture on students’ critical thinking. The Review of Higher Education, 23(4), 421-441. http://doi.org/10.1353/rhe.2000.0020
77. Turkish Language Assosiation (n.d.). Ability. In Updated Turkish Dictionary. Retrieved February 28, 2021. https://www.tdk.gov.tr/
78. Usta, H.G. (2020). Sınav kaygı ölçeği maddelerinin çeşitli yöntemlere göre değişen madde fonksiyonlarının incelenmesi [Analysis of Test Anxiety Scale items in terms of differential item functioning by different methods]. Cumhuriyet International Journal of Education, 9(4), 1225-1242. https://doi.org/10.30703/cije.703337
79. Valenzuela, J., Nieto, A.M., & Saiz, C. (2011). Critical thinking motivational scale: A contribution to the study of relationship between critical thinking and motivation. Journal of Research in Educational Psychology, 9(2), 823 848. http://repositorio.ual.es/bitstream/handle/10835/819/Art_24_588.pdf?sequence=1
80. Yıldırım, H., & Büyüköztürk, Ş. (2018). Using the Delphi Technique and focus-group interviews to determine item bias on the Mathematics Section of the Level Determination Exam for 2012. Educational Sciences: Theory & Practice, 18(2), 447-470. http://doi.org/10.12738/estp.2018.2.0317