Comparison of Classification Judgments Scaling Methods

Abstract

In this study, 264 performance tasks 6, 7 and 8th grade secondary school students answered will be examined by the scaling method with classification criteria to determine whether classification judgements differ or not obtained from the scores of 0-15 by four raters with analytical and holistic rubrics (“0-3” beginning level, “4-6” developable, “7-9” apprentices, “10-12” headworker, “13-15” master). The research is based on descriptive research with the dimension of not aiming generalization and  revealing the present situation. In the study, data were collected with two performance tasks, including evidence of validity and reliability and rubrics related to these tasks. As a result of the research, it is seen that scale values and orders of classification judgements of four raters for 264 performances in terms of problem solving ability can be partially changed according to rubric type and the method used. While the scaling method does not cause any difference in the order of scale values in the case of using  holistic rubric; it is seen that the rank order of the scale obtained from the B full data matrix and the scale sequences obtained from the B rank numerical solution and D state full matrices are different in the case of using analytical rubric.

Keywords

• Rubric,Scaling,Judgmental decisions scaling,Classification judgments

References

1. Alharby, E. R. (2006). A comparison between two scoring methods, holistic vs analytic, using two measurement models, the generalizability theory and the many-facet rasch measurement, within the context of performance assessment. Phd Dissertation. Pennsylvenia State University.
2. Andrade, H. G. (2001). The effects of instructional rubrics on learning to write. Current Issues in Education, 4(4). Retrieved from http://cie.asu.edu/ojs/index.php/cieatasu/article/view/1630
3. Anıl, D. ve Güler. N (2006). İkili karşılaştırma yöntemi ile ölçekleme çalışmasına bir örnek. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 30, 30-36.
4. Atılgan, H., Kan, A. ve Doğan, N. (2009). Eğitimde ölçme ve değerlendirme. Ankara: Anı Yayıncılık.
5. Bauer, B. A. (1981). A study of the reliabilities and cost-efficiencies of three methods of assessment for writing ability. (ERIC Document Reproduction Service No. ED 216357).
6. Bal, Ö. (2011). Seviye belirleme sınavı (SBS) başarısında etkili olduğu düşünülen faktörlerin sıralama yargıları kanunuyla ölçeklenmesi. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 2(2), 200-209.
7. Boring, R. L. (2002). Human and computerized essay assessment: a comparative analysis of holistic, analytic and latent semantic methods. Unpublished Thesis, Department of Psychology, New Mexico State University, Las Cruses, New Mexico.
8. Brualdi, A. (1998). Implementing performance assessment in the classroom. Practical Assessment, Research & Evaluation, 6(2). Retrieved from http://pareonline.net/getvn.asp?v=6&n=2

9. Ekinci, A., Bindak, R. ve Yıldırım, M.C. (2012). Ilköğretim okulu yöneticilerinin öğretmenlerin mesleki sorunlarına empatik yaklaşımlarının ikili karşılaştırmalar metodu ile incelenmesi. Gaziantep Üniversitesi Sosyal Bilimler Dergisi, 11(3), 759 -776. ISSN: 1303-0094
10. Follman, J. C. & Anderson, J. A. (1967). An investigation of reliability of five procedures for grading english themes. Research in the Teaching of English, 1, 190-200.
11. Güler, N., Anıl, D. (2009). Scaling through pair-wise comparison method in required characteristics of students applying for post graduate programs. International Journal of Human Sciences [Online]. 6(1), 627-639.
12. Haladyna, T. M. (1997). Writing test ıtems to evaluate higher order thinking. USA: Allyn & Bacon.
13. Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research & Evaluation, 7(25). Retrieved from http://pareonline.net/getvn.asp?v=7&n=25
14. Moskal, B. M. (2000). Scoring rubrics: what, when and how?. Practical Assessment, Research & Evaluation, 7(3). Retrieved from http://pareonline.net/getvn.asp?v=7&n=3
15. Nartgün, Z, (2006). Öğretmenlik meslek bilgisi derslerinin önem düzeyinin ikili karşılaştırmalarla ölçeklenmesi. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 6(2), 161-176.
16. National Council of Teachers of Mathematics. (2000). Assessment standard for school mathematics. Reston, Va. NCTM (Available online document). Retrieved from http://standards.nctm.org
17. Kan, A. (2008). Yargıcı kararlarına dayalı ölçekleme yöntemlerinin karşılaştırılması üzerine ampirik bir çalışma. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 35: 186-194.
18. Klein, S. P., Stecher, B. M., Shavelson, R. J., McCaffrey, D., Ormseth, T., Bell, R.M., Comfort, K., & Othman, A. R. (1998). Analytic versus holistic scoring of science performance tasks. Applied Measurement in Education, 11, 121-137.
19. Linn, R.L. (1993). Educational assessment: expanded expectations and challenges. Educational Evaluation and Policy Analysis. 15, 1-16. doi: 10.3102/01623737015001001
20. Öğretmen, T. (2008). Alan tercih envanteri: ölçeklenmesi, geçerlik ve güvenirliği. Türk Eğitim Bilimleri Dergisi, 6(3), 507-522.
21. Özer, Y. ve Acar, M. (2011). Öğretmenlik mesleği genel yeterlikleri üzerine ikili karşılaştırma yöntemiyle bir ölçekleme çalışması. Çukurova Üniversitesi Eğitim Fakültesi Dergisi. 3, 89-101.
22. Öztürk, N., Özdemir, S ve Gelbal, S. (2011). Iki farklı ölçekleme yaklaşımından elde edilen ölçek değerlerinin tutarlığının incelenmesi. Yayımlanmamış Bildiri. 20. Ulusal Eğitim Bilimleri Kurultayı, Burdur.
23. Resnick, L. B. (1987). Education and learning to think. Washington, D.C.: National Academy. Retrieved from http://www.nap.edu/openbook.php?record_id=1032&page=1
24. Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.
25. Turgut, M. F. ve Baykul, Y. (1992). Ölçekleme teknikleri. Ankara: ÖSYM Yayınları.
26. Torgerson, W. S. (1958). Theory and methods of scaling. Newyork: John Wiley & Sons Inc.
27. Vrasidas, C. (2000). Constructivism versus objectivism: implications for interaction, course design, and evaluation in distance education. International Journal of Educational Telecommunications, 6(4), 339-362.
28. Wiggins, G. (1989). Teaching to the (authentic) test. Educational Leadership, 46 (7), 41–47.
29. Zollman, A. & Jones, D. L. (1994). Accommodating assessment and learning: utilizing portfolios in teacher education with preservice teachers. Paper presented at the annual meeting of the Research Council on Diagnostic and Prescritive Mathematics, Texas, IL