Investigating Differential Item Functioning of Ankara University Examination for Foreign Students by Recursive Partitioning Analysis in the Rasch Model

Abstract

This study aims to determine whether items in the Ankara University Examination for Foreign Students Basic Learning Skills Test function differently according to country and gender using the Recursive Partitioning Analysis in the Rasch Model. The variables used in the recursive partitioning of the data are country and gender. The population of the study is composed of 2476 individuals. Since the study includes comparisons across countries, the country is accepted as a criterion in determining the sample group. Thus, the sample of the study consists of 615 individuals selected from Azerbaijan, Bulgaria, and Syria. To investigate differential item functioning (DIF) of the items of the test, the Rasch tree method was used. As a result of the analysis, DIF has been detected in 16 items at the 0.001 significance level. However, these items have been identified to have similar difficulty parameters in all countries. Finally, items have not shown DIF according to gender.

Keywords

• Differential Item Functioning
• Rrecursive Partitioning Analysis
• Rasch Tree Method
• Examinations for International Students

Investigating Differential Item Functioning of Ankara University Examination for Foreign Students by Recursive Partitioning Analysis in the Rasch Model

Öz

This study aims to determine whether items in the Ankara University Examination for Foreign Students Basic Learning Skills Test function differently according to country and gender using the Recursive Partitioning Analysis in the Rasch Model. The variables used in the recursive partitioning of the data are country and gender. The population of the study is composed of 2476 individuals. Since the study includes comparisons across countries, the country is accepted as a criterion in determining the sample group. Thus, the sample of the study consists of 615 individuals selected from Azerbaijan, Bulgaria, and Syria. To investigate differential item functioning (DIF) of the items of the test, the Rasch tree method was used. As a result of the analysis, DIF has been detected in 16 items at the 0.001 significance level. However, these items have been identified to have similar difficulty parameters in all countries. Finally, items have not shown DIF according to gender.

Anahtar Kelimeler

• Differential Item Functioning
• Rrecursive Partitioning Analysis
• Rasch Tree Method
• Examinations for International Students

References

1. Altıntaş, Ö. (2016). Ankara Üniversitesi Yabancı Uyruklu Öğrenci Seçme Testinin Ölçme Değişmezliğinin Örtük Sınıf ve Rasch Modeline Göre İncelenmesi [Investigating The Measurement Invariance of Ankara University Foreign Student Selection Test by Latent Class and Rasch Model]. PhD diss., Ankara University.
2. AERA., APA., & NCME. (2014). Standards for Educational and Psychological Testing. Washington: American Educational Research Association.
3. Anastasi, A. (1979). Fields of Applied Psychology. 2nd ed. New York: McGraw-Hill Book Company.
4. Anastasi, A., & Urbina, S. (1997). Psychological Testing. 7th ed. New Jersey: Prentice-Hall International, Inc.
5. Ankara University. (2013). Ankara Üniversitesine Yurtdışından Öğrenci Kabulüne İlişkin Yönerge. Ankara Üniversitesi Senato Kararı Örneği [Instruction on Admission of Foreign Students to Ankara University. Ankara University Senate Decision Example]. Decision Date: 23.06.2013. Number of Meetings: 365. Number of Decisions: 3100.
6. ANKUDEM. (2011). Ankara University Examination for Foreign Student Selection and Placement Exam (AYOS) Project Final Report. Project No: 11Y5250001. Ankara: Ankara University Scientific Research Projects Scientific Coordinator.
7. Asimov, I. 2004. Asimov’s Chronology of Science and Discovery: Updated and Illustrated. Norwalk: The Easton Press.
8. Best, J. W. (1970). Research in Education. 2nd ed. New Jersey: Prentice-Hall Inc.

9. Brown, R. T., Reynolds, C. R., & Whitaker, J. S. (1999). Bias in mental testing since Bias in Mental Testing. School Psychology Quarterly, 14(3), 208-238. DOI: http://dx.doi.org/10.1037/h0089007
10. Büyüköztürk, Ş., Akgün, Ö. E., Demirel, F., Karadeniz, Ş., & Kılıç Çakmak, E. (2015). Bilimsel Araştırma Yöntemleri (Scientific Research Methods). Ankara: Pegem Academy Publishing Co.
11. Camilli, G., & Shepard, L. A. (1994). Methods for Identifying Biased Test Items. California: Sage Publication, Inc.
12. Carman, C. A., & Taylor, D. K. (2010). Socioeconomic Status Effects on Using the Naglieri Nonverbal Ability Test (NNAT) to Identify the Gifted/Talented. Gifted Child Quarterly, 54(2), 75–84. DOI: https://doi.org/10.1177/0016986209355976
13. Cohen, R. J., Montague, P., Nathanson, L. S., & Swerdlik M. E. (1988). Psychological Testing: An Introduction to Tests and Measurement. California: Mayfield Publishing Company.
14. Crocker, L., & Algina, J. (1986). Introduction to Classical Modern Test Theory. New York: Harcourt Brace Jovanovich College Publishers.
15. Cronbach, L. J. (1971). Test Validation. In Educational Measurement. 2nd ed., edited by R. L. Thorndike, 443-507. Washington: American Council on Education.
16. Cronbach, L. J. (1990). Essentials of Psychological Testing. 5th ed. New York: Harper and Collins Publishers, Inc.
17. Cronbach, L. J., & Meehl, P. E. (1955). Classics in the History of Psychology. Psychological Bulletin, 52, 281 302. Retrieved from https://psychclassics.yorku.ca/Cronbach/construct.htm
18. Hambleton, R. K. (2002). Adapting Achievement Tests into Multiple Languages for International Assessments. In Methodological Advances in Cross-National Surveys of Educational Achievement, edited by A. C. Porter and A. Gamoran, 58-79. Washington: National Academy Press.
19. Hambleton, R. K., Merenda, P. F., & Spielberger, C. D., eds. (2005). Adapting Educational and Psychological Tests for Cross-cultural Assessment. New Jersey: Lawrence Erlbaum Associates, Inc.
20. Karakaya, İ., & Kutlu, Ö. (2012). An Investigation of Item Bias in Turkish Sub Tests in Level Determination Exam. Education and Science, 37(165), 2-15. Retrieved from http://egitimvebilim.ted.org.tr/index.php/EB/article/view/1342/433
21. Karasar, N. (2015). Bilimsel Araştırma Yöntemi [Scientific Research Method]. 28th ed. Ankara: Nobel Academy Publishing Co.
22. Kopf, J. (2013). Model-based Recursive Partitioning Meets Item Response Theory: New Statistical Methods for the Detection of Differential Item Functioning and Appropriate Anchor Selection. PhD diss., Munich: Ludwig-Maximilians-University Department of Statistics. Retrieved from https://edoc.ub.uni-muenchen.de/16434/1/Kopf_Julia.pdf
23. Kopf, J., Augustin, T., & Strobl, C. (2010). The Potential of Model-Based Recursive Partitioning in the Social Sciences: Revisiting Ockham’s Razor. Technical report number 88. Munich: University of Munich Department of Statistics. Retrieved from https://pdfs.semanticscholar.org/c8ee/da16de9cb040066e5cb64aa37ceb58493286.pdf
24. Kutlu, Ö., & Karakaya, İ. (2007). Orta Öğretim Kurumları Öğrenci Seçme ve Yerleştirme Sınavının Faktör Yapılarına İlişkin Bir Araştırma. [A Research on the Factor Structure of Secondary Education Institutions’ Student Selection and Placement Test]. Elementary Education Online, 6(3), 397 410. Retrieved from http://dergipark.gov.tr/download/article-file/90996
25. Linn, R. L., ed. (1989). Educational Measurement. 3rd ed. New Jersey: American Council on Education and Macmillan Publishing Company.
26. Linn, R. L., & Gronlund N. E. (2000). Measurement and Assessment in Teaching. 8th ed. New Jersey: Prentice-Hall International, Inc.
27. Lohman, D. F., & Lakin, J. M. (2011). Reasoning and Intelligence. In Handbook of Intelligence, edited by R. J. Sternberg, & S. B. Kaufman, 419-441. New York: Cambridge University Press.
28. Maller, S. J. (2001). Differential Item Functioning in the WISC-III: Item Parameters for Boys and Girls in the National Standardization Sample. Educational and Psychological Measurement, 61(5), 793–817. DOI: https://doi.org/10.1177/00131640121971527
29. Messick, S. (1989). Validity. In Educational Measurement. 3rd ed., edited by R. L. Linn, 13-103. New Jersey: American Council on Education and Macmillan Publishing Company.
30. Oral, T. (1985). Lise Başarı Ölçüleri ile ÖSYS Puanları Arasındaki Uyum [Concordance between High School Success Metrics and University Entrance Exam (ÖSYS) Scores]. PhD diss., Hacettepe University.
31. Osterlind, S. J., & Everson, H. T. (2009). Differential Item Functioning. California: Sage Publications, Inc.
32. Öner, N. (1987). Kültürlerarası Ölçek Uyarlamasında Bir Yöntembilim Modeli [A Methodology Model in Intercultural Scale Adaptation]. Turkish Journal of Psychology, 6(21), 80-83.
33. R Core Team. (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
34. Resing, W. C. M. (2005). Intelligence Testing. In Encyclopedia of Social Measurement, edited by K. Kempf-Leonard, 307-315. San Diego: Elsevier Academic Press.
35. Reynolds, C. R., & Suzuki, L. (2013). Bias in Psychological Assessment: An Empirical Review and Recommendations. In Handbook of Psychology Vol. 10: Assessment Psychology. 2nd ed., edited by J. R. Graham, J. A. Naglieri, & I. B. Weiner, 82-113. New Jersey: John Wiley and Sons, Inc.
36. Shepard, L., Camilli, G., & Averill, M. (1981). Comparison of Procedures for Detecting Test-Item Bias with Both Internal and External Ability Criteria. Journal of Educational and Behavioral Statistics, 6(4), 317-375. DOI: https://doi.org/10.3102/10769986006004317
37. Sternberg, R. J. (1997). The Concept of Intelligence and Its Role in Lifelong Learning and Success. American Psychologist, 52(10), 1030-1037. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.826.5234&rep=rep1&type=pdf
38. Strobl, C., Kopf, J., & Zeileis, A. (2015). Rasch Trees: A new method for detecting differential item functioning in the Rasch model. Psychometrika, 80(2), 289-316. Retrieved from https://eeecon.uibk.ac.at/~zeileis/papers/Strobl+Kopf+Zeileis-2015.pdf
39. Thornell, J. G., & McCoy, A. (1985). The Predictive Validity of The Graduate Record Examination for Subgroups of Students in Different Academic Disciplines. Educational and Psychological Measurement, 45(2), 415 419. DOI: http://dx.doi.org/10.1177/001316448504500229
40. Toffler, A. (1980). The Third Wave. New York: McGraw-Hill Book Company.
41. Toker, F. (1997). Türkiye’de Yükseköğretime Giriş [Entrance to Higher Education in Turkey]. Ankara: ÖSYM Publications.
42. Toivainen, T., Papageorgiou, K. A., Tostoc, M. G., & Kovas Y. (2017). Sex Differences in Non-verbal and Verbal Abilities in Childhood and Adolescence. Intelligence, 64, 81-88. DOI: https://doi.org/10.1016/j.intell.2017.07.007
43. Turgut, M. F. & Baykul, Y. (1992-1). Ölçekleme Teknikleri [Scaling Methods]. Ankara: ÖSYM Publications.
44. USYM. (1978). İki Aşamalı Üniversitelerarası Seçme ve Yerleştirme Sınavının Temel İlkeleri [Basic Principles of Two-Stage University Selection and Placement Exam]. Ankara: ÖSYM Publications.
45. USYM. (1980a). İki Aşamalı Üniversitelerarası Seçme ve Yerleştirme Sistemi [Two-Stage University Selection and Placement System]. KY-02-80-0001. Ankara: ÖSYM Publications.
46. USYM. (1980b). İki Aşamalı Üniversitelerarası Seçme ve Yerleştirme Sistemi [Two-Stage University Selection and Placement System]. KY-02-80-0002. Ankara: ÖSYM Publications.
47. Walsh, W. B., & Betz, N. E. (1995). Tests and Assessment. 3rd ed. Englewood Cliffs. New Jersey: Prentice-Hall International, Inc.
48. Wicherts, J. M. (2007). Group Differences in Intelligence Test Performance. PhD diss., University of Amsterdam. Retrieved from https://pure.uva.nl/ws/files/4175964/46967_Wicherts.pdf
49. YOK. (2013). Yurtdışından Öğrenci Kabulüne İlişkin Esaslar [Principles for the Acceptance of Students From Abroad]. The decision of the General Council of Higher Education. Decision Date: 01.02.2013.
50. Zeileis, A., Hothorn, T., & Hornik, K. (2008). Model-based recursive partitioning. Journal of Computational and Graphical Statistics. 17(2), 492-514.
51. Zeileis, A., Strobl, C., Wickelmaier, F., & Kopf, J. (2011). psychotree: Recursive partitioning based on psychometric models. R package version 0.12-1, Retrieved from http://CRAN.R-project.org/package=psychotree
52. Zwick, R., & Sklar, J. C. (2005). Predicting College Grades and Degree Completion Using High School Grades and SAT Scores: The Role of Student Ethnicity and First Language. American Educational Research Journal, 42, 439 464. DOI: https://doi.org/10.3102/00028312042003439