17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS' LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY

Abstract

Nowadays, rapid changes in science and technology increase the demand of qualified individuals who have signs of disciplined mind which is highlighted in Howard Gardner’s five minds as one type of mind. So, it is important to measure whether individuals have disciplined mind or not. Based on this idea, it is aimed to evaluate the reliability of rubric used in determining the seventh grade students’ levels of disciplined mind in terms of generalizability theory in this research. It is possible in the Generalizability theory to calculate reliability both for the relative decisions (G coefficient) and the absolute decisions (Φ coefficient). The G and the Φ coefficients calculated with six tasks and three raters in this research were found as .86 and .79, respectively. In the generalizability study, attempts are made to predict the situations where the error can be reduced to the minimum for specific purposes through the decision study. According to decision study results, it was found that increasing the number of raters would not bring any benefits to the similar studies to be performed in the future; and it would not be a practical way in cases where it is difficult to find competent raters

Keywords

• disciplined mind, reliability, generalizability theory, rubric

17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS' LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY

Abstract

Günümüzde, bilim ve teknolojideki hızlı değişim Howard Gardner'ın beş zihin alanından birisi olarak vurguladığı disiplinli zihne sahip nitelikteki insana olan ihtiyacı arttırmaktadır. Bu nedenle bireylerin disiplinli zihne sahip olup olmadığını belirlemek önemlidir. Bu bağlamda, bu araştırmada yedinci sınıf öğrencilerinin disiplinli zihin özelliklerini belirlemek için kullanılan dereceli puanlama anahtarı güvenirliğinin Genellenebilirlik Kuramı ile kestirilmesi amaçlanmıştır. Genellenebilirlik kuramı ile göreli kararlar (G katsayısı) ve mutlak kararlar (Φ katsayısı) için güvenirlik katsayısının hesaplanması mümkün olup; bu çalışmada elde edilen puanların G ve Φ katsayısı, altı görev ve üç puanlayıcı için sırasıyla .86 ve .79 olarak bulunmuştur. Genellenebilirlik kuramında ayrıca, karar (K) çalışmaları yoluyla değişkenlik kaynaklarının sayısının değişmesi durumunda güvenirliğin tahmin edilmesi de mümkün olmaktadır. Bu araştırmadaki karar çalışması sonuçları, gelecekte yapılacak benzer nitelikteki çalışmalarda, puanlayıcı sayısını arttırmanın puanların güvenirliğini arttırmaya katkı sağlamayacağını göstermektedir. Nitelikli puanlayıcı bulmanın kolay olmadığı durumlarda, puanlayıcı sayısını arttırmanın uygulanma açısından da pratik olmayacağı düşünüldüğünde; daha fazla puanlayıcı ile benzer çalışmalar yapılmasının önerilemeyeceği sonucuna varılmıştır

Keywords

• -

References

1. Alharby, E. R. (2006). A comparison between two scoring methods, holistic vs. analytic using two measurement models, the generalizability theory and the many facet rasch measurement within the context of performance assessment (Unpublished
2. Doctoral Dissertation). Pennsylvania State University. Allal, L., & Cardinet, J. (1997). Generalizability theory. In J.P. Keeres (Ed.), Educational research, methodology, and measurement: An international handbook (2nd, pp. 737- 741). Cambridge, United Kindom: Cambridge University.
3. Baker, E. L., Abedi, J., Linn, R. L., & Niemi, D. (1995). Dimensionality and generalizability of domain-independent performance assessments. Journal of educational research, 89, 197–205.
4. Baykul, Y. (2000). Measurement in education and psychology: Classical test theory and its application. Ankara: ÖSYM, Turkey.
5. Brennan, R. L. (1992). Elements of generalizability theory. New York: Springer-Verlog.
6. Brennan, R. L. (2001). Generalizability theory. Iowa: ACT Publications.
7. Chun, M. (2010). Taking teaching to (performance) task: Linking pedagogical and assessment practice. Retrieved from http://www.learningace.com/doc/2844170/7d60cfc1e1ecc6db4a2df9e4e4038863/chun _change_takingteachingtotask.
8. Crocker, L., Algina, J. (1986). Introduction to classical and modern test theory. USA: Harcourt Brace Javanovich College Publishers.

9. Cronbach, J. L., Gleser, G. C., Nanda & Rajaratman, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley and Sons.
10. Gardner, H. (2006). Five minds for the future. USA: Harvard Business School Press.
11. Goodwin, L. D.&Goodwin, W. L. (1991). Research notes: Using generalizability theory in early childhood special education.
12. Journal of Early Intervention, 15 (2), 193-204. Goodwin, L. D. (2001). Interrater agreement and reliability. Measurement in Psychical Education and Exercises Science, 5(1), 13
13. Güler, N. (2009). Generalizability theory and comparison of the results of g and d studies computed by SPSS and GENOVA packet programs. Education and Science, 34, 154, 93-103.
14. Güler, N. (2011). The comparison of the reliability according to generalizability theory and classical test theory on random data.
15. Education and Science, 36, 162, 225-234. Güler, N., Uyanık, G. K., & Teker, G. T. (2012). Generalizability theory. Ankara, Turkey: Pegem Akademi Publishing.
16. Güler, N. & Gelbal, S. (2010). Studying Reliability of Open Ended Mathematics Items According to the Classical Test Theory and Generalizability Theory. Educational Sciences: Theory & Practice. 10, 2, 989-1019.
17. Hsu, L. (2012). Applications of generalizability theory to estimate the reliability o EFL learners’ performance-based assessment: a preliminary study. Educational Research, 3(2), 145-154.
18. Kieffer, K. M. (1998). Why generalizability theory is essential and classical test theory is often inadequate. Paper presented at the annual meeting of the Southwestern Psychological Association. New Orleans, LA. USA.
19. Lane, S., Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
20. Lee, G.& Frisbie, D. A. (1999). Estimating reliability under a generalizability theory model for test scores composed of testlets.
21. Applied Measurement in Education, 12(3), 237-255. Lee, G.& Fitzpatrick, A. R. (2003). The effects of a student sampling plan on estimates of the errors for stuents passing rates.
22. Journal of Educational Measurement, 40(1), 17-28. Lei, P., Smith, M., & Suen, H. K. (2007). The use of generalizability theory to estimate data reliability in single subject observational research. Psychology in Schools, 44(5), 433-439.
23. Lucas, T., Brian, L., Arnetz, J. & Arnetz, B. (2010). Do ratings of african-american cultural competency reflect characteristics of providers or perceivers? Initial demonstration of a generalizability theory approach. Psychology, Health & Medicine, 15(4), 445- 4
24. Mercer, S. H., Dufrene, B. A., Martell, K. Z., Harpole, L. L., Mitchell, R. R. & Blaze, J. J. (2012). Generalizability theory analysis of CBM maze reliability in third- through fifth- grade students. Assessments for Effective Intervention, 20(10), 1-8.
25. Mushquash, C. & O’Connor, B. P. (2006). SPSS and SAS programs for generalizability theory analysis. Behavior Research Methods, 38 (3), 542-547.
26. Shavelson, R. J. & Webb, N. M. (1991). Generalizability theory: A Primer. USA: Sage Publications.
27. Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2005). A comparison of generalizability theory and many facet measurement in an analysis of college sophomore writing. Assessing Writing, 9, 239-261.
28. Volpe, R. J., McConaughy, S. H., & Hintze, J. M. (2009). Generalizability of classroom behavior problem and on-task scores from the direct observation form. School Psychology Review, 38,3.
29. Yelboğa, A. & Tavşancıl, E. (2010). The examination of reliability according to classical test and generalizability on a job performance scale. Educational Sciences: Theory & Practices, 10(3), 1847-1854.
30. Yin, Y. & Shavelson, R. J. (2008). Application of generalizability theory to concept map assessment research. Applied
31. Measurement in Education, 21, 273-291.