BibTex RIS Cite

17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS' LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY

Year 2014, Issue: 42, 0 - 0, 20.06.2015

Özge Can Aran Neşe Güler Nuray Senemoğlu

Abstract

Nowadays, rapid changes in science and technology increase the demand of qualified individuals who have signs of disciplined mind which is highlighted in Howard Gardner’s five minds as one type of mind. So, it is important to measure whether individuals have disciplined mind or not. Based on this idea, it is aimed to evaluate the reliability of rubric used in determining the seventh grade students’ levels of disciplined mind in terms of generalizability theory in this research. It is possible in the Generalizability theory to calculate reliability both for the relative decisions (G coefficient) and the absolute decisions (Φ coefficient). The G and the Φ coefficients calculated with six tasks and three raters in this research were found as .86 and .79, respectively. In the generalizability study, attempts are made to predict the situations where the error can be reduced to the minimum for specific purposes through the decision study. According to decision study results, it was found that increasing the number of raters would not bring any benefits to the similar studies to be performed in the future; and it would not be a practical way in cases where it is difficult to find competent raters

Keywords

disciplined mind reliability generalizability theory rubric

References

  • Alharby, E. R. (2006). A comparison between two scoring methods, holistic vs. analytic using two measurement models, the generalizability theory and the many facet rasch measurement within the context of performance assessment (Unpublished
  • Doctoral Dissertation). Pennsylvania State University. Allal, L., & Cardinet, J. (1997). Generalizability theory. In J.P. Keeres (Ed.), Educational research, methodology, and measurement: An international handbook (2nd, pp. 737- 741). Cambridge, United Kindom: Cambridge University.
  • Baker, E. L., Abedi, J., Linn, R. L., & Niemi, D. (1995). Dimensionality and generalizability of domain-independent performance assessments. Journal of educational research, 89, 197–205.
  • Baykul, Y. (2000). Measurement in education and psychology: Classical test theory and its application. Ankara: ÖSYM, Turkey.
  • Brennan, R. L. (1992). Elements of generalizability theory. New York: Springer-Verlog.
  • Brennan, R. L. (2001). Generalizability theory. Iowa: ACT Publications.
  • Chun, M. (2010). Taking teaching to (performance) task: Linking pedagogical and assessment practice. Retrieved from http://www.learningace.com/doc/2844170/7d60cfc1e1ecc6db4a2df9e4e4038863/chun _change_takingteachingtotask.
  • Crocker, L., Algina, J. (1986). Introduction to classical and modern test theory. USA: Harcourt Brace Javanovich College Publishers.
  • Cronbach, J. L., Gleser, G. C., Nanda & Rajaratman, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley and Sons.
  • Gardner, H. (2006). Five minds for the future. USA: Harvard Business School Press.
  • Goodwin, L. D.&Goodwin, W. L. (1991). Research notes: Using generalizability theory in early childhood special education.
  • Journal of Early Intervention, 15 (2), 193-204. Goodwin, L. D. (2001). Interrater agreement and reliability. Measurement in Psychical Education and Exercises Science, 5(1), 13
  • Güler, N. (2009). Generalizability theory and comparison of the results of g and d studies computed by SPSS and GENOVA packet programs. Education and Science, 34, 154, 93-103.
  • Güler, N. (2011). The comparison of the reliability according to generalizability theory and classical test theory on random data.
  • Education and Science, 36, 162, 225-234. Güler, N., Uyanık, G. K., & Teker, G. T. (2012). Generalizability theory. Ankara, Turkey: Pegem Akademi Publishing.
  • Güler, N. & Gelbal, S. (2010). Studying Reliability of Open Ended Mathematics Items According to the Classical Test Theory and Generalizability Theory. Educational Sciences: Theory & Practice. 10, 2, 989-1019.
  • Hsu, L. (2012). Applications of generalizability theory to estimate the reliability o EFL learners’ performance-based assessment: a preliminary study. Educational Research, 3(2), 145-154.
  • Kieffer, K. M. (1998). Why generalizability theory is essential and classical test theory is often inadequate. Paper presented at the annual meeting of the Southwestern Psychological Association. New Orleans, LA. USA.
  • Lane, S., Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
  • Lee, G.& Frisbie, D. A. (1999). Estimating reliability under a generalizability theory model for test scores composed of testlets.
  • Applied Measurement in Education, 12(3), 237-255. Lee, G.& Fitzpatrick, A. R. (2003). The effects of a student sampling plan on estimates of the errors for stuents passing rates.
  • Journal of Educational Measurement, 40(1), 17-28. Lei, P., Smith, M., & Suen, H. K. (2007). The use of generalizability theory to estimate data reliability in single subject observational research. Psychology in Schools, 44(5), 433-439.
  • Lucas, T., Brian, L., Arnetz, J. & Arnetz, B. (2010). Do ratings of african-american cultural competency reflect characteristics of providers or perceivers? Initial demonstration of a generalizability theory approach. Psychology, Health & Medicine, 15(4), 445- 4
  • Mercer, S. H., Dufrene, B. A., Martell, K. Z., Harpole, L. L., Mitchell, R. R. & Blaze, J. J. (2012). Generalizability theory analysis of CBM maze reliability in third- through fifth- grade students. Assessments for Effective Intervention, 20(10), 1-8.
  • Mushquash, C. & O’Connor, B. P. (2006). SPSS and SAS programs for generalizability theory analysis. Behavior Research Methods, 38 (3), 542-547.
  • Shavelson, R. J. & Webb, N. M. (1991). Generalizability theory: A Primer. USA: Sage Publications.
  • Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2005). A comparison of generalizability theory and many facet measurement in an analysis of college sophomore writing. Assessing Writing, 9, 239-261.
  • Volpe, R. J., McConaughy, S. H., & Hintze, J. M. (2009). Generalizability of classroom behavior problem and on-task scores from the direct observation form. School Psychology Review, 38,3.
  • Yelboğa, A. & Tavşancıl, E. (2010). The examination of reliability according to classical test and generalizability on a job performance scale. Educational Sciences: Theory & Practices, 10(3), 1847-1854.
  • Yin, Y. & Shavelson, R. J. (2008). Application of generalizability theory to concept map assessment research. Applied
  • Measurement in Education, 21, 273-291.

17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS' LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY

Year 2014, Issue: 42, 0 - 0, 20.06.2015

Özge Can Aran Neşe Güler Nuray Senemoğlu

Abstract

Günümüzde, bilim ve teknolojideki hızlı değişim Howard Gardner'ın beş zihin alanından birisi olarak vurguladığı disiplinli zihne sahip nitelikteki insana olan ihtiyacı arttırmaktadır. Bu nedenle bireylerin disiplinli zihne sahip olup olmadığını belirlemek önemlidir. Bu bağlamda, bu araştırmada yedinci sınıf öğrencilerinin disiplinli zihin özelliklerini belirlemek için kullanılan dereceli puanlama anahtarı güvenirliğinin Genellenebilirlik Kuramı ile kestirilmesi amaçlanmıştır. Genellenebilirlik kuramı ile göreli kararlar (G katsayısı) ve mutlak kararlar (Φ katsayısı) için güvenirlik katsayısının hesaplanması mümkün olup; bu çalışmada elde edilen puanların G ve Φ katsayısı, altı görev ve üç puanlayıcı için sırasıyla .86 ve .79 olarak bulunmuştur. Genellenebilirlik kuramında ayrıca, karar (K) çalışmaları yoluyla değişkenlik kaynaklarının sayısının değişmesi durumunda güvenirliğin tahmin edilmesi de mümkün olmaktadır. Bu araştırmadaki karar çalışması sonuçları, gelecekte yapılacak benzer nitelikteki çalışmalarda, puanlayıcı sayısını arttırmanın puanların güvenirliğini arttırmaya katkı sağlamayacağını göstermektedir. Nitelikli puanlayıcı bulmanın kolay olmadığı durumlarda, puanlayıcı sayısını arttırmanın uygulanma açısından da pratik olmayacağı düşünüldüğünde; daha fazla puanlayıcı ile benzer çalışmalar yapılmasının önerilemeyeceği sonucuna varılmıştır

Keywords

-

References

  • Alharby, E. R. (2006). A comparison between two scoring methods, holistic vs. analytic using two measurement models, the generalizability theory and the many facet rasch measurement within the context of performance assessment (Unpublished
  • Doctoral Dissertation). Pennsylvania State University. Allal, L., & Cardinet, J. (1997). Generalizability theory. In J.P. Keeres (Ed.), Educational research, methodology, and measurement: An international handbook (2nd, pp. 737- 741). Cambridge, United Kindom: Cambridge University.
  • Baker, E. L., Abedi, J., Linn, R. L., & Niemi, D. (1995). Dimensionality and generalizability of domain-independent performance assessments. Journal of educational research, 89, 197–205.
  • Baykul, Y. (2000). Measurement in education and psychology: Classical test theory and its application. Ankara: ÖSYM, Turkey.
  • Brennan, R. L. (1992). Elements of generalizability theory. New York: Springer-Verlog.
  • Brennan, R. L. (2001). Generalizability theory. Iowa: ACT Publications.
  • Chun, M. (2010). Taking teaching to (performance) task: Linking pedagogical and assessment practice. Retrieved from http://www.learningace.com/doc/2844170/7d60cfc1e1ecc6db4a2df9e4e4038863/chun _change_takingteachingtotask.
  • Crocker, L., Algina, J. (1986). Introduction to classical and modern test theory. USA: Harcourt Brace Javanovich College Publishers.
  • Cronbach, J. L., Gleser, G. C., Nanda & Rajaratman, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley and Sons.
  • Gardner, H. (2006). Five minds for the future. USA: Harvard Business School Press.
  • Goodwin, L. D.&Goodwin, W. L. (1991). Research notes: Using generalizability theory in early childhood special education.
  • Journal of Early Intervention, 15 (2), 193-204. Goodwin, L. D. (2001). Interrater agreement and reliability. Measurement in Psychical Education and Exercises Science, 5(1), 13
  • Güler, N. (2009). Generalizability theory and comparison of the results of g and d studies computed by SPSS and GENOVA packet programs. Education and Science, 34, 154, 93-103.
  • Güler, N. (2011). The comparison of the reliability according to generalizability theory and classical test theory on random data.
  • Education and Science, 36, 162, 225-234. Güler, N., Uyanık, G. K., & Teker, G. T. (2012). Generalizability theory. Ankara, Turkey: Pegem Akademi Publishing.
  • Güler, N. & Gelbal, S. (2010). Studying Reliability of Open Ended Mathematics Items According to the Classical Test Theory and Generalizability Theory. Educational Sciences: Theory & Practice. 10, 2, 989-1019.
  • Hsu, L. (2012). Applications of generalizability theory to estimate the reliability o EFL learners’ performance-based assessment: a preliminary study. Educational Research, 3(2), 145-154.
  • Kieffer, K. M. (1998). Why generalizability theory is essential and classical test theory is often inadequate. Paper presented at the annual meeting of the Southwestern Psychological Association. New Orleans, LA. USA.
  • Lane, S., Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
  • Lee, G.& Frisbie, D. A. (1999). Estimating reliability under a generalizability theory model for test scores composed of testlets.
  • Applied Measurement in Education, 12(3), 237-255. Lee, G.& Fitzpatrick, A. R. (2003). The effects of a student sampling plan on estimates of the errors for stuents passing rates.
  • Journal of Educational Measurement, 40(1), 17-28. Lei, P., Smith, M., & Suen, H. K. (2007). The use of generalizability theory to estimate data reliability in single subject observational research. Psychology in Schools, 44(5), 433-439.
  • Lucas, T., Brian, L., Arnetz, J. & Arnetz, B. (2010). Do ratings of african-american cultural competency reflect characteristics of providers or perceivers? Initial demonstration of a generalizability theory approach. Psychology, Health & Medicine, 15(4), 445- 4
  • Mercer, S. H., Dufrene, B. A., Martell, K. Z., Harpole, L. L., Mitchell, R. R. & Blaze, J. J. (2012). Generalizability theory analysis of CBM maze reliability in third- through fifth- grade students. Assessments for Effective Intervention, 20(10), 1-8.
  • Mushquash, C. & O’Connor, B. P. (2006). SPSS and SAS programs for generalizability theory analysis. Behavior Research Methods, 38 (3), 542-547.
  • Shavelson, R. J. & Webb, N. M. (1991). Generalizability theory: A Primer. USA: Sage Publications.
  • Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2005). A comparison of generalizability theory and many facet measurement in an analysis of college sophomore writing. Assessing Writing, 9, 239-261.
  • Volpe, R. J., McConaughy, S. H., & Hintze, J. M. (2009). Generalizability of classroom behavior problem and on-task scores from the direct observation form. School Psychology Review, 38,3.
  • Yelboğa, A. & Tavşancıl, E. (2010). The examination of reliability according to classical test and generalizability on a job performance scale. Educational Sciences: Theory & Practices, 10(3), 1847-1854.
  • Yin, Y. & Shavelson, R. J. (2008). Application of generalizability theory to concept map assessment research. Applied
  • Measurement in Education, 21, 273-291.
There are 31 citations in total.

Details

Primary Language Turkish
Journal Section Articles
Authors

Özge Can Aran This is me

Neşe Güler This is me

Nuray Senemoğlu This is me

Publication Date June 20, 2015
Published in Issue Year 2014 Issue: 42

Cite

APA Aran, Ö. C., Güler, N., & Senemoğlu, N. (2015). 17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi(42).
AMA Aran ÖC, Güler N, Senemoğlu N. 17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi. June 2015;(42).
Chicago Aran, Özge Can, Neşe Güler, and Nuray Senemoğlu. “17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY”. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi, no. 42 (June 2015).
EndNote Aran ÖC, Güler N, Senemoğlu N (June 1, 2015) 17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi 42
IEEE Ö. C. Aran, N. Güler, and N. Senemoğlu, “17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY”, Dumlupınar Üniversitesi Sosyal Bilimler Dergisi, no. 42, June 2015.
ISNAD Aran, Özge Can et al. “17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY”. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi 42 (June 2015).
JAMA Aran ÖC, Güler N, Senemoğlu N. 17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi. 2015.
MLA Aran, Özge Can et al. “17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY”. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi, no. 42, 2015.
Vancouver Aran ÖC, Güler N, Senemoğlu N. 17)AN EVALUATION OF THE RUBRIC USED IN DETERMINING STUDENTS’ LEVELS OF DISCIPLINED MIND IN TERMS OF GENERALIZABILITY THEORY. Dumlupınar Üniversitesi Sosyal Bilimler Dergisi. 2015(42).

Dergimiz EBSCOhost, ULAKBİM/Sosyal Bilimler Veri Tabanında, SOBİAD ve Türk Eğitim İndeksi'nde yer alan uluslararası hakemli bir dergidir.