Analytical Weighting Scoring for Physics Multiple Correct Items to Improve the Accuracy of Students’ Ability Assessment

Abstract

Purpose: This is a developmental research study that aims to develop a model of polytomous scoring based-on weighting for multiple correct items in the subject of physics. Weighting was analytically applied based on question complexity and imposed penalties on wrong answers. Research Methods: Within the development model, Fenrich's development cycle, consisting of analysis, planning, design, development, implementation, evaluation, and revision, was selected throughout the cycle. The multiple correct items used have 3–4 options. The items were implemented to 140 upper secondary school students and 410 first-year undergraduate students. The students’ physics ability was analyzed using the Quest program, and the results of dichotomous and polytomous scoring were compared. Findings: The results of this study showed that the analytical weighting scoring based on a complexity and penalty system on the developed assessment items generated a higher number of scoring level categories (three to seven categories) than that of dichotomous scoring (only two categories), estimated students’ physics abilities more accurately and in greater detail, had an approximate distribution closer to the normal distribution, and produced a standard deviation smaller than that of dichotomous scoring. Thus, the analytical weighting scoring for multiple correct items in this study was able to produce a more accurate estimation of physics ability than those using dichotomous scoring. Implications for Research and Practice: It is recommended that the assessment of physics ability using multiple-correct items on a large scale can apply the analytical weighting scoring based on the complexity of the content and a penalty system.

Keywords

• Analytical weighting scoring,accuracy of estimation,physics aptitude

References

1. Adams, R.J., & Khoo, S.T. (1996). Quest (Computer software). The interactive test analysis system. Victoria: Acer.
2. Adeyemo, S. A. (2010). Students’ ability level and their competence in problem-solving task in physics. International Journal of Educational Research and Technology, 1(2), 35 – 47.
3. Ali, S. H., Carr, P. A., & Ruit, K. G. (2016). Validity and reliability of scores obtained on multiple-choice questions: Why functioning distractors matter. Journal of the Scholarship of Teaching and Learning, 16 (1), 1-14.
4. Baghaei, P., & Dourakhshan, A. (2016). Properties of single-response and double-response multiple-choice grammar items. International Journal of Language Testing, 6 (1), 33-49.
5. Baker, J.G., Rounds, J.B., & Zeron, M.A. (2000). A comparison of graded response and rasch partial credit models with subjective well-being. Journal of Educational and Behavioral Statistic, 25(3), 253-270.
6. Bishara, A. J., & Lanzo, L. A. (2015). All of the above: When multiple correct response options enhance the testing effect. Journal Memory, 23(7), 1013-1028.
7. Bush, M. (2015). Reducing the need for guesswork in multiple-choice tests. Assessment & Evaluation in Higher Education, 40(2), 218-231.
8. Bond, T. G., & Fox, C. M. (2007). Applying the rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah: Lawrence Erlbaum Associates, Publishers.

9. Donoghue, J. R. (2005). An empirical examination of the IRT information of polythomously scored reading items under the generalized PCM. Journal of Educational Measurement, 31(4), 295-311.
10. Emaikwu, S. O. (2015). Recent issues in the construction and scoring of multiple-choice items in examinations. The International Journal of Humanities & Social Studies. 3(6), 201-207.
11. Fenrich, P. (2004). Instructional design tips for virtually teaching practical skills. In Proceedings of the 2004 Informing Science and IT Education Joint Conference. Rockhampton, Australia June 25–28, 2004.
12. Frisbie, D.A. (1992). The multiple true-false format: A status review. Educational Measurement: Issues and Practice, 11(4), 21-26
13. Grunert, M. L., Raker, J. R., Murphy, K. L., & Holme, T. A. (2013). Polytomous versus dichotomous scoring on multiple-choice examinations: development of a rubric for rating partial credit. Journal of Chemical Education, 90(10), 1310-1315.
14. Haladyna, T.M., Downing, S.M., & Rodriguez, M.C. (2002). A review of multiple choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15 (3), 309-334.
15. Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. London: Sage Publications.
16. Hickson, S., Reed, W. R., & Sander, N. (2012). Estimating the effect on grades of using multiple-choice versus constructive-response questions: Data from the classroom. Educational Assessment, 17(4), 200-213.
17. Holt, A. (2006). An analysis of negative marking in multiple-choice assessment. The 19th Annual Conference of the National Advisory Committee on Computing Qualifications (NACCQ 2006), Wellington, New Zealand. Samuel Mann and Noel Bridgeman (Eds)
18. Jiao, H., Liu, J., Hainie, K., Woo, A., & Gorham, J. (2012). Comparison between dichotomous and polytomous scoring of innovative items in a large-scale computerized adaptive test. Educational and Psychological Measurement, 72(3), 493–509.
19. King, K. V., Gardner, D. A., Sasha Zucker, S., & Jorgensen, M. A. (2004). The distractor rationale taxonomy: Enhancing multiple-choice items in reading and mathematics. Assessment Report, Pearson Education, Inc, July 2004.
20. Klein, P., Müller, A., & Kuhn, J. (2017). Assessment of representational competence in kinematics. Physical Review Physics Education Research 13, 1-18.
21. Lin, C.J. (2008). Comparisons between classical test theory and item response theory in automated assembly of parallel test form. The Journal of Technology, Learning, and Assessment. 6(8), 1-42.
22. Martin, D. L., & Itter, D. (2014).Valuing assessment in teacher education-multiple-choice competency testing. Australian Journal of Teacher Education, 39(7), 1-14.
23. Merrell, J. D., Cirillo, P. F., Schwartz, P. M., & Jeffrey A. Webb, J. A. (2015). Multiple-choice testing using immediate feedback—assessment technique (IF AT) forms: Second-chance guessing vs. second-chance learning. Higher Education Studies. 5(5), 50-55.
24. Oosterhof, A. (2003). Developing and using classroom assessments (3th ed). Upper Saddle River: Merrill Prentice Hall. Redish, E.F., Scherr, R.E. & Tuminaro, J. (2006). Reverse engineering the solution of a “simple” physics problem: Why learning physics is harder than it looks. The Physics Teacher, 44(5), 293-300.
25. Rodriguez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, Summer, 3-13.
26. Stankous, N. V. (2016). Constructive response vs. multiple-choice tests in math: American experience and discussion (Review). European Scientific Journal. May 2016, 308-3016.
27. Tognolini, J., & Davidson, M. (Juli 2003). How do we operationalise what we value? Some technical chalenges in assessing higher order thinking skills. Paper presented in the Natinaonal Roundtable on Assessment Conference. Darwin, Australia.
28. Wiseman, C. S. (2012). A comparison of the performance of analytic vs. holistic scoring rubrics to assess L2 writing, Iranian Journal of Language Testing, 2 (1), 59-92.
29. Wooten, M. M., Cool, A. M., Edward E. Prather, E. E., & Tanner, K. D. (2014). Comparison of performance on multiple-choice questions and open-ended questions in an introductory astronomy laboratory. Physical Review Special Topics - Physics Education Research, 10, 1-22. Wright, B.D., & Masters, G.N. (1982). Rating scale analysis. Chicago: Mesa Press.
30. Wu, B.C. (2003). Scoring multiple true-false items: A comparison of sumed scores and response pattern scores at item and test level. Research report, Educational Resources International Center (ERIC) USA, 1-40