Three, four, and none of the above options in multiple-choice items

Abstract

High-quality multiple-choice (MC) items are essential for creating efficient, valid assessments. Haladyna, Downing, and Rodriguez (2002) suggested that using plausible distractors is crucial to achieving this goal, although distractor creation can be time-consuming and challenging. Haladyna et al. (2002) provided two related test development guidelines: #18, “Write as many plausible distractors as you can,” and #25, “Use carefully None of the above.” This research aims to test the impact of these two guidelines on item difficulty (p), item discrimination (r), and test reliability for mathematics items empirically. The research findings have revealed that item discrimination and test reliability were not statistically different across MC items with four options, three options, and NOTA options while the means of item difficulty of four-option MC items was not statistically different from those of three-option and NOTA-option MC items. However, the mean of item difficulty of NOTA-option MC items was statistically lower than those of three-option.

Keywords

• assessment
• item-writing guidelines
• multiple-choice items
• number of options
• none of the above

Çoktan seçmeli sorularda üç, dört ve hiçbiri seçenekleri

Öz

Etkili ve geçerli değerlendirmeler yapabilmek için yüksek kaliteli çoktan seçmeli sorular (maddeler) oluşturmak gereklidir. Bu amaç doğrultusunda Haladyna, Downing, and Rodriguez (2002) çoktan seçmeli madde oluşturma sürecinde uygun çeldiricilerin kullanılmasının önemli olduğunu vurgulamaktadırlar. Ancak bu çeldirilerin oluşturulması zaman alıcı ve zahmetli bir süreçtir. Bu bağlamda Haladyna ve diğerleri (2002) test geliştirme ile ilişkili olan "Olabildiği kadar uygun ve mantıklı çeldiricileri yazınız" ve "Hiçbiri seçeneğini dikkatli kullanınız" ilkelerini rapor etmektedirler. Bu araştırmanın amacı bu iki ilke kullanımının matematik sorularınn zorluk derecesine, ayırt edicilik derecesine ve test güvenirliğine etkisini test etmektir. Araştırma sonucunda, madde ayırt edicilik derecesi ve test güvenirliği dört seçenekli, üç seçenekli, ve “hiçbiri” seçeneğine sahip maddeler arasında istatistiksel olarak anlamlı bir farklılık göstermediği bulunmuştur. Benzer şekilde, dört seçenekli maddelerin madde zorluk dereceleri, üç seçenekli ve “hiçbiri” seçeneğine sahip maddelerin madde zorluk derecelerinden istatistiksel olarak anlamlı bir farklılık göstermediği bulunurken, “hiçbiri” seçeneğine sahip maddelerin madde zorluk dereceleri üç seçenekli maddelerin madde zorluk derecelerinden istatistiksel olarak düşük olduğu elde edilmiştir.

Anahtar Kelimeler

• Değerlendirme
• soru-yazma teknikleri
• çoktan seçmeli sorular
• seçenek sayısı
• yukarıdakilerden hiçbiri

References

1. Abad, F. J., Olea, J., & Ponsoda, V. (2001). Analysis of the optimum number of alternatives from the Item Response Theory. Psicothema, 13(1), 152-158.
2. Baghaei, P., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple choice items. Psychological Test and Assessment Modeling, 53(2), 192-211.
3. Burton, S.J., Sudweeks, R. R., Merrill, P. F., & Wood, B. (1991). How to prepare better multiple-choice test items: Guidelines for university faculty. Brigham Young University Department of Instructional Science. Retrieved from http://testing.byu.edu/info/handbooks/betteritems.pdf
4. Cizek, G. J., & O'Day, D. M. (1994). Further investigation of nonfunctioning options in multiple-choice test items. Educational and Psychological Measurement, 54(4), 861–872. doi:10.1177/0013164494054004002
5. Common Core State Standards Initiative (CCSSI). (2010). Common Core state standards for mathematics. Washington, DC: National Governors Association Center for Best Practices and the Council of Chief State School Officers.
6. Crehan, K. D., & Haladyna, T. M. (1991).The validity of two item-writing rules. The Journal of Experimental Education, 59(2), 183–192. doi:10.1080/00220973.1991.10806560
7. Crehan, K. D., Haladyna, T. M., & Brewer, B. W. (1993). Use of an inclusive option and the optimal number of options for multiple-choice items. Educational and Psychological Measurement, 53(1), 241-247. doi:10.1177/0013164493053001027
8. Dehnad, A., Nasser, H., & Hosseini, A. F. (2014). A comparison between three-and four-option multiple choice questions. Procedia-Social and Behavioral Sciences, 98, 398–403. doi:10.1016/j.sbspro.2014.03.432

9. Delgado, A. R., & Prieto, G. (1998). Further evidence favoring three-option items in multiple-choice tests. European Journal of Psychological Assessment, 14(3), 197–201. doi:10.1027/1015-5759.14.3.197
10. DiBattista, D., Sinnige-Egger, J., & Fortuna, G. (2014). The “none of the above” option in multiple-choice testing: An experimental study. The Journal of Experimental Education, 82(2), 168-183. doi:10.1080/00220973.2013.795127
11. Downing, S. M. (2005). The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Advances in health sciences education, 10(2), 133-143.doi:10.1007/s10459-004-4019-5
12. Duhachek, A., & Iacobucci, D. (2004). Alpha's standard error (ASE): an accurate and precise confidence interval estimate. Journal of Applied Psychology, 89(5), 792–808. doi:10.1037/0021-9010.89.5.792
13. Frary, R. B. (1991). The none-of-the-above option: An empirical study. Applied Measurement in Education, 4(2), 115–124. doi:10.1207/s15324818ame0402_2
14. Frey, B.B., Petersen, S., Edwards, L. M., Pedrotti, J.T., & Peyton, V. (2005). Item-writing rules: Collective wisdom. Teaching and Teacher Education, 21(4), 357–364.doi:10.1016/j.tate.2005.01.008
15. Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 37–50. doi:10.1207/s15324818ame0201_3
16. Haladyna, T. M., & Downing, S. M. (1993). How many options is enough for a multiple-choice test item? Educational and psychological measurement, 53(4), 999–1010.doi:10.1177/0013164493053004013
17. Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choiceitem-writing guidelines for classroom assessment. Applied Measurement in Education,15(3), 309–334. doi:10.1207/S15324818AME1503_5
18. Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York, NY: Routledge.
19. Hansen, J. D., & Dexter, L. (1997). Quality multiple-choice test questions: Item-writing guidelines and an analysis of auditing testbanks. Journal of Education for Business 73(2), 94–97. doi:10.1080/08832329709601623
20. IBM Corp. Released 2012. IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corp.
21. Kingston, N.M., & Kramer, L.B. (2013). Highstakes test construction and test use. In T.D. Little (Ed.),The Oxford handbook of quantitative methods: Vol. 1 foundations (pp. 189–205). New York, NY: Oxford University Press.
22. Kingston, N.M., Scheuring, S.T., & Kramer, L.B. (2013). Test Development Strategies. In Kurt Geisinger (Ed.) APA Handbook of Testing and Assessment in Psychology. Washington, DC: APA Books.
23. Knowles, S. L.,&Welch, C. A. (1992). A meta-analytic review of item discrimination and difficulty in multiple-choice items using "none-of-the-above.” Educational and Psychological Measurement, 52(3), 571–577. doi:10.1177/0013164492052003006
24. Kolstad, R. K., & Kolstad, R. A. (1991). The option “none of these” improves multiple-choice test items. Journal of Dental Education, 55(2), 161–163.
25. Landrum, R. E., Cashin, J. R., & Theis, K. S. (1993). More evidence in favor of three-option multiple-choice tests. Educational and Psychological Measurement, 53(3), 771–778. doi:10.1177/0013164493053003021
26. Lee, H., & Winke, P. (2013). The differences among three-, four-, and five-option-item formats in the context of a high-stakes English-language listening test. Language Testing, 30(1), 99-123.
27. McCoubrie, P. (2004). Improving the fairness of multiple-choice questions: a literature review. Medical Teacher, 26(8), 709–712. doi:10.1080/01421590400013495
28. Odegard, T. N., & Koen, J. D. (2007). “None of the above” as a correct and incorrect alternative on a multiple-choice test: Implications for the testing effect. Memory, 15(8), 873–885. doi:10.1080/09658210701746621
29. Rich, C. E., & Johanson, G. A. (1990, April).An item-level analysis of “none of the above.”Paper presented at the annual meeting of the American Educational Research Association, Boston, MA. Retrieved from http://files.eric.ed.gov/fulltext/ED400299.pdf
30. Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysisof 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13. doi:0.1111/j.1745-3992.2005.00006.x
31. Rogers, W. T., & Harley, D. (1999). An empirical comparison of three-and four-choice items and tests: Susceptibility to testwiseness and internal consistency reliability. Educational and Psychological Measurement, 59(2), 234–247. doi:10.1177/00131649921969820
32. StataCorp (2013). Stata: release 13 - statistical software. College Station, TX: StataCorp LP
33. Shizuka, T., Takeuchi, O., Yashima, T. & Yoshizawa, K. (2006). A comparison of three-and four-option English tests for university entrance selection purposes in Japan. Language Testing, 23(1), 35–57. doi:10.1191/0265532206lt319oa
34. Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis.BMC Medical Education, 9(1), 40. doi:10.1186/1472-6920-9-40
35. Tarrant, M., & Ware, J. (2010). A comparison of the psychometric properties of three-and four-option multiple-choice questions in nursing assessments. Nurse Education Today, 30(6), 539–543. doi:10.1016/j.nedt.2009.11.002
36. Tollefson, N. (1987). A comparison of the item difficulty and item discrimination of multiple-choice items using the"none of the above" and one correct response options. Educational and Psychological Measurement, 47(2), 377–383.doi:10.1177/0013164487472010
37. Trevisan, M. S., Sax, G., & Michael, W. B. (1991). The effects of the number of options per item and student ability on test validity and reliability. Educational and Psychological Measurement, 51(4),829–837. doi:10.1177/001316449105100404