The Effect of Presenting Geometry Items with and without Shapes on the Psychometric Properties of the Test and Students’ Test Scores

Abstract

This research intended to examine the effects of presenting geometry items with and without shapes on the psychometric properties of the test and students’ test scores. The study was conducted on 480 eighth grade students. Within the scope of the study, two geometry tests were crafted, one with shapes and the other without shapes. Both tests consisted of 15 multiple-choice items. In the data collection process, a counterbalanced design was followed and the two tests were administered to the students three weeks apart. Analyses were carried out on 405 students who participated in both applications and whose test forms could be matched. The factor analysis results revealed that the factor loadings of the items and extracted variance were higher for the test with shapes compared to the shape-free form. The Cronbach’s alpha coefficient of the test containing shapes was found to be significantly higher than that calculated for the test without shapes. According to item difficulties, the questions with shapes were easier for the students than the shape-free questions. In terms of discrimination indices a difference in favor of the shape-containing test was observed in almost all items. Ferguson delta statistic, which is a measure of discrimination for the overall test, was higher in the shape-contaninig test. Correlation analysis denoted a strong positive relationship between student scores on the two tests. The paired sample t-test proved that there was a statistically significant difference between students’ scores on the shape-containing and shape-free tests. These results indicate that the geometry tests with and without shapes differ in terms of both psychometric properties and students’ test scores.

Keywords

• Shape-containing geometry items
• geometry items without shapes
• psychometric properties

References

1. Abd El-Mohsen, M. M. (2008). The effect of stem length in multiple choice questions on item difficulty in syllabus-based vocabulary test items difficulty in syllabus-based vocabulary test items [Unpublished Master Theses, The American University in Cairo]. Retrieved from https://fount.aucegypt.edu/retro_etds/2195/
2. Ascalon, M. E., Meyers, L. S., Davis, B. W., & Smits, N. (2007). Distractor similarity and item-stem structure: Effects on item difficulty. Applied Measurement in Education, 20(2), 153–170. https://doi.org/10.1080/08957340701301272
3. Atalmış, E. H. (2018). The use of three-option multiple choice items for classroom assessment. International Journal of Assessment Tools in Education, 5(2), 314–324. https://doi.org/10.21449/ijate.421167
4. Atalmış, E. H., & Kingston, N. (2017). Three, four, and none of the above options in multiple-choice items. Turkish Journal of Education, 6(4), 143–157. https://doi.org/10.19128/turje.333687
5. Atalmış, E. H., & Kingston, N. M. (2018). The impact of homogeneity of answer choices on item difficulty and discrimination. Sage Open, 8(1). https://doi.org/10.1177/2158244018758147
6. Aybek, E. C. (2021). Data preparation for factor analysis. https://shiny.eptlab.com/dp2fa/
7. Aydın, E., Kertil, M., Yılmaz, K., & Topçu, T. (2006). Examining contextual support in geometry learning in terms of student and question level [Geometri öğreniminde bağlamsal desteğin öğrenci ve soru seviyesi açısından incelenmesi] [Full text oral presentation]. Ⅶ. National Science and Mathematics Education Congress, Gazi University, Gazi Education Faculty, Ankara.
8. Bacon, D. R. (2003). Assessing learning outcomes: A comparison of multiple-choice and short-answer questions in a marketing context. Journal of Marketing Education, 25(1), 31–36. https://doi.org/10.1177/0273475302250570

9. Baghaei, P., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple choice items. Psychological Test and Assessment Modeling, 53(2), 192–211.
10. Barut, M. E. O., & Retnawati, H. (2020). Geometry learning in vocational high school: Investigating the students’ difficulties and levels of thinking. Journal of Physics: Conference Series 1613(1), 012058. https://doi.org/10.1088/17426596/1613/1/012058
11. Bishara, A. J., & Lanzo, L. A. (2014). All of the above: When multiple correct response options enhance the testing effect. Memory, 23(7), 1013–1028. https://doi.org/10.1080/09658211.2014.946425
12. Brooks, G. P., & Johanson, G. A. (2003). TAP: Test Analysis Program. Applied Psychological Measurement, 27(4), 303–304. https://doi.org/10.1177/0146621603027004007
13. Büyüköztürk, Ş. (2010). A manual of data analysis for social sciences [Sosyal bilimler için veri analizi el kitabı] (11. ed). Pegem Academy.
14. Cheng, H. (2004). A comparison of multiple-choice and open-ended response formats for the assessment of listening proficiency in English. Foreign Language Annals, 37(4), 544–553. https://doi.org/10.1111/j.1944-9720.2004.tb02421.x
15. Cizek, G. J. (1994). The effect of altering the position of options in a multiple-choice examination. Educational and Psychological Measurement, 54(1), 8–20. https://doi.org/10.1177/0013164494054001002
16. Cohen, J. (1977). Statistical power analysis for the behavioral sciences. Academic.
17. Corriero, E. (2017). Counterbalancing. In The SAGE Encyclopedia of Communication Research Methods (Vol. 4, pp. 278–281). Sage. https://doi.org/10.4135/9781483381411
18. Crehan, K. D., Haladyna, T. M., & Brewer, B. W. (1993). Use of an inclusive option and the optimal number of options for multiple-choice items. Educational and Psychological Measurement, 53(1), 241–247. https://doi.org/10.1177/0013164493053001027
19. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. HarcourtBrace Jovanovich.
20. Çetin, B., & Türkan A., (2013). The effect of the compatibility and incompatibility of the shapes with their actual values ​​in secondary school 8th grade geometry test questions on the psychometric properties of the test [İlköğretim 8. sınıf geometri testi sorularında şekillerin gerçek değerlerine uygun çizilmesiyle, farklı çizilmesinin testin psikometrik özelliklerine etkisi]. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi. 4(2), 52–63. https://doi.org/10.21031/epod.77190
21. Çiftçi, O., & İşleyen, T. (2022). Üçgenin açıortayları ve kenarortayları konusunda öğrencilerin karşılaştıkları öğrenme güçlükleri. Ahi Evran Üniversitesi Kırşehir Eğitim Fakültesi Dergisi, 23(Özel Sayı), 509–560. https://doi.org/10.29299/kefad.943663
22. Demir, E. (2010). Uluslararası öğrenci değerlendirme programı (PISA) bilişsel alan testlerinde yer alan soru tiplerine göre Türkiye’de öğrenci (Tez No. 257803), [Yüksek lisans tezi, Hacettepe Üniversitesi]. YÖK Ulusal Tez Merkezi.
23. Diedenhofen, B., & Musch, J. (2016). cocron: A web interface and R package for the statistical comparison of Cronbach’s alpha coefficients. International Journal of Internet Science, 11(1), 51–60.
24. Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Prentice Hall.
25. Erkuş, A. (2012). Measurement and scale development in psychology–I: Basic concepts and procedures [Psikolojide ölçme ve ölçek geliştirme-1: Temel kavramlar ve işlemler]. Pegem Academy.
26. Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1), 93–103. https://doi.org/10.1177/014662168701100107
27. Ferguson, G. A. (1949). On the theory of test discrimination. Psychometrika, 14(1), 61–68. https://doi.org/10.1007/bf02290141
28. Field, A. (2013). Discovering statistics using SPSS (3rd ed.). Sage.
29. Goldfarb, R. (2021). Consuming and producing research in communication sciences and disorders: Developing power of professor. Plural.
30. Graveter, F. J., & Forzano, L. B. (2018). Research methods for the behavioral sciences (6th ed.). Cengage.
31. Gültekin, S., & Demirtaşlı, N. Ç. (2012). Comparing the test information obtained through multiple choice, open-ended and mixed item tests based on item response theory. Elementary Education Online, 11(1), 251–263.
32. Haladyna, T. M., & Downing, S. M. (1993). How many options is enough for a multiple-choice test item? Educational and Psychological Measurement, 53(4), 999–1010. https://doi.org/10.1177/0013164493053004013
33. Harasym, P. H., Doran, M. L., & Brant, R., & Lorscheider, F.L. (1993). Negation in stems of single-response multiple-choice items: An overestimation of student ability. Evaluation & the Health Professions, 16(3), 342–357. https://doi.org/10.1177/016327879301600307
34. Harasym, P. H., Price, P. G., Brant, R., Violato, C., Lorscheider, F. L. (1992). Evaluation of negation in stems of multiple-choice items. Evaluation & the Health Professions, 15(2), 198–220. https://doi.org/10.1177/016327879201500205
35. Hernandez, E. & Zalava, G. (2017). Accurate items for inaccurate in undergraduate physics students. In M.S. Ramírez- Montoya (Eds.), Handbook of research on driving STEM learning with educational technologies (pp. 315-340). IGI Global.
36. Hohensinn, C., & Baghaei, P. (2017). Does the position of response options in multiple-choice tests matter? Psicológica, 38(1), 93–109.
37. İlhan, M., Boztunç Öztürk, N., & Şahin, M. G. (2020). The effect of the item’s type and cognitive level on its difficulty index: The sample of TIMSS 2015. Participatory Educational Research, 7(2), 47–59. https://doi.org/10.17275/per.20.19.7.2
38. JASP Team (2022). JASP (Version 0.18.1.0) [Computer software]. https://jasp-stats.org/
39. Jonsdottir, A. H., Jonmundsson, T., Armann, I. H., Gunnarsdottir, B. B., & Stefansson, G. (2021, 8-9 March). The effect of the number of distractors and the “none of the above” – “all of the above” options in multiple choice questions [Conference presentation]. 5th International Technology, Education and Development Conference. https://doi.org/10.21125/inted.2021.1540
40. Kan, A., Bulut, O., & Cormier, D. C. (2019). The impact of item stem format on the dimensional structure of mathematics assessments. Educational Assessment, 24(1), 13–32. https://doi.org/10.1080/10627197.2018.1545569
41. Karanfil, T., & Neufeld, S. (2020). The role of order and sequence of options in multiple-choice questions for high-stakes tests of English language proficiency. International Journal of Applied Linguistics and English Literature, 9(6), 110–129. https://doi.org/10.7575/aiac.ijalel.v.9n.6p.110
42. Karpuz, Y., Koparan, T., & Güven, B. (2014). Students’ use of shape and concept knowledge in geometry [Geometride öğrencilerin şekil ve kavram bilgisi]. Turkish Journal of Computer and Mathematics Education, 5(2), 108–118. https://dergipark.org.tr/en/pub/turkbilmat/issue/21573/231505
43. Kline, P. (1993). Handbook of psychological testing (2nd ed.). Routledge.
44. Koepf, T. M. (2018). The effect of item stem and response option length on the item analysis outcomes of a career and technical education multiple choice assessment [Unpublished Doctoral Dissertation, Western Michigan University]. Retrieved from https://scholarworks.wmich.edu/dissertations/3366/
45. Lindquist, E. F. (1936). The theory of test construction. In H. W. Hawkes, E. F. Linquist & C. R. Mann (Eds.), The construction and use of achievement examinations: A manual for secondary school teachers (pp. 17–106). Houghton Mifflin.
46. Lions, S., Dartnell, P., Toledo, G., Godoy, M. I., Córdova, N., Jiménez, D., & Lemarié, J. (2023). Position of correct option and distractors impacts responses to multiple-choice items: Evidence from a national test. Educational and Psychological Measurement, 83(5), 861–884. https://doi.org/10.1177/00131644221132335
47. Lions, S., Monsalve, C., Dartnell, P., Godoy, M. I., Córdova, N., Jiménez, D., Blanco, M. P., Ortega, G., & Lemarié, J. (2021). The position of distractors in multiple-choice test items: The strongest precede the weakest. Frontiers in Educiton, 6, 731763. https://doi.org/10.3389/feduc.2021.731763
48. Michael–Chrysanthou, P., Panaoura, A., & Gagatsis, A. (2024). Exploring secondary school students’ geometrical figure apprehension: cognitive structure and levels of geometrical ability. Educational Studies in Mathematics. https://doi.org/10.1007/s10649-024-10317-5
49. Ministry of National Education of Türkiye Republic. (2018). Central examination for secondary education institutions that will accept students by test: Numerical part. Retrieved from https://odsgm.meb.gov.tr/meb_iys_dosyalar/2018_06/03153730_SAYISAL_BYLYM_A_kitapYY.pdf
50. Mitchell, J. E., Crosby, R. D., Wonderlich, S., & Adson, D. E. (2000). Elements of clinical research in psychiatry. American Psychiatric.
51. Nwadinigwe, P. I., & Naibi, L. (2013). The number of options in a multiple-choice test ıtem and the psychometric characteristics. Journal of Education and Practice, 4(28), 189–196. Retrieved from https://www.iiste.org/Journals/index.php/JEP/article/view/9944
52. Öksüz, Y., & Güven Demir, E. (2019). Comparison of open ended questions and multiple choice tests in terms of psychometric features and student performance. Hacettepe University Journal of Education, 34(1), 259–282. https://doi.org/10.16986/HUJE.2018040550
53. Özer Özkan, Y., & Özaslan, N. (2018). Student achievement in Turkey, according to question types used in PISA 2003-2012 mathematic literacy tests. International Journal of Evaluation and Research in Education (IJERE), 7(1), 57–64. https://pdfs.semanticscholar.org/7e84/37899e70c78fbe2dde7ab179ccca7eb6a0a0.pdf
54. Paler-Calmorin, L., & Calmorin, M. A. (2007). Research methods and thesis writing (2nd ed.). Rex Book Store.
55. Pallant, J. (2005). SPSS survival manual: A step by step guide to data analysis using SPSS for Windows (Version 12). Allen & Unwin.
56. Raymond, M. R., Stevens, C., & Bucak, S. D. (2019). The optimal number of options for multiple-choice questions on high-stakes tests: Application of a revised index for detecting nonfunctional distractors. Advances in Health Science Education, 24, 141–150. https://doi.org/10.1007/s10459-018-9855-9
57. Rodriguez, M. C. (2002). Choosing an item format. In G. Tindal & T. M. Haladyna (Eds.), Large-scale assessment programs for all students: Validity, technical adequacy, and implementation (pp. 213–231). Lawrence Erlbaum Associates.
58. Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3–13. https://doi.org/10.1111/j.1745-3992.2005.00006.x
59. Salkind, N. J. (2010). Statistics for people who (think they) hate statistics (3rd ed.). Sage.
60. Schaefer, J. M. L. (2009). The effects of stem completeness and stem orientation on multiple-choice item difficulty and discrimination [Unpublished Master Theses, California State University]. Retrieved from https://hdl.handle.net/10211.9/162
61. Shin, J., Bulut, O., & Gierl, M. J. (2019). The effect of the most-attractive-distractor location on multiple-choice item difficulty. The Journal of Experimental Education, 88(4), 643–659. https://doi.org/10.1080/00220973.2019.1629577
62. Şahin, M. D. (2022). Exploratory factor analysis [Açımlayıcı faktör analizi]. In S. Göçer Şahin & M. Buluş (Eds.), Applied statistics step by step [Adım adım uygulamalı istatistik] (pp. 309–342) Pegem Academy.
63. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Pearson.
64. Temizkan, M., & Sallabaş, M. E. (2015). Comparison of multiple choice tests and open-ended questions in the assessment of reading comprehension skills [Okuduğunu anlama becerisinin değerlendirilmesinde çoktan seçmeli testlerle açık uçlu yazılı yoklamaların karşılaştırılması]. Dumlupınar University Journal of Socıal Sciences, 30, 207–220.
65. Terranova, C. (1969). The effects of negative stems in multiple-choice test items. Unpublished doctoral dissertation, State University of New York at Buffalo. (30, 2390A).
66. Vegada, B., Shukla, A., Khilnani, A., Charan, J., & Desai, C. (2016). Comparison between three option, four option and five option multiple choice question tests for quality parameters: A randomized study. Indian Journal of Pharmacology, 48(5), 571–575. https://doi.org/10.4103/0253-7613.190757
67. Violato, C. (1991). Item difficulty and discrimination as a function of stem completeness. Psychological Reports, 69(3), 739–743. https://doi.org/10.2466/pr0.1991.69.3.739
68. Violato, C., & Harasym, P. H. (1987). Effects of structural characteristics of stem format of multiple-choice items on item difficulty and discrimination. Psychological Reports, 60(3_part_2), 1259–1262. https://doi.org/10.1177/0033294187060003-251.1
69. Violato, C., & Marini, A. E. (1989). Effects of stem orientation and completeness of multiple-choice items on item difficulty and discrimination. Educational and Psychological Measurement, 49(1), 287–295. https://doi.org/10.1177/0013164489491032
70. Yılmaz Koğar, E., & Soysal, S. (2023). Examination of response time effort in TIMSS 2019: Comparison of Singapore and Türkiye. International Journal of Assessment Tools in Education, 10(Special Issue), 174–193. https://doi.org/10.21449/ijate.1343248
71. Yılmaz, S. (2007). Misconceptions of second-degree primary school’s students about problem solving (Thesis Number. 200688). [Master Thesis, Eskişehir Osmangazi University], Eskişehir.
72. Zhang, F., & Lidbury, B. A. (2013). Evaluating a genetics concepts inventory. In F. Zhang (Eds.), Sustainable language support practices in science education: Technologies and solutions (pp. 116–128). Medical Information Scince Reference.
73. Zulaiha, R., Dian Rahdiani, F., Rahman, A., & Al Anfal, M. F. (2021). Analysis of difficulty level and discriminating power between multiple choices and essay items on math test. Advances in Social Science, Education and Humanities Research, 545, 62–68. https://doi.org/10.2991/assehr.k.210423.065