Reliability of Criterion-Dependent Measurement Tools According To Generalizability Theory: Application in The Case of Eating Skills

Abstract

Applied behavioral analysis is one of the most frequently utilized to help students with mental disabilities develop skills of living independently. Applied behavior analysis is based on direct measurement with criterion-dependent measurement tools. As applications of them have played significant role to evaluate students with mental disabilities; the reliability of these measures has become increasingly important issue. For this reason, in this study, generalizability theory was used to estimate the reliability of them and the role of different sources of error in the variability of measurements in the case of measuring eating by the spoon skills. The results indicated measurement results varied depending on occasions, occasion by task and task by rater effects whereas effects of tasks and raters were negligible.

Keywords

• Reliability
• Generalizability Theory
• Criterion-Dependent Measurement Tools

GENELLENEBİLİRLİK KURAMINA GÖRE ÖLÇÜT BAĞIMLI ÖLÇME ARAÇLARINDA GÜVENİRLİK: YEMEK YEME BECERİLERİ ÖRNEĞİNDE BİR UYGULAMA

Abstract

Zihinsel engelli öğrencilere bağımsız yaşama becerileri kazandırmanın en etkili yollarından birisi uygulamalı davranış analizidir. Uygulamalı davranış analizinin temelini ölçüt bağımlı ölçme araçlarıyla yapılan doğrudan ölçümler oluşturmaktadır. Ölçüt bağımlı ölçme araçlarının uygulanması zihinsel engelli öğrencilerin değerlendirilmesinde önemli rol oynadığı için bu ölçme araçlarının güvenirlikleri gittikçe önemli bir konu olmaktadır. Bu nedenle bu araştırmada bu ölçme araçlarının güvenirliği ve çeşitli hata kaynaklarının ölçüm sonuçlarının değişkenliğinde oynadığı rol, kaşıkla yemek yeme becerilerinin ölçülmesi örneğinde genellenebilirlik kuramı aracılığıyla kestirilmeye çalışılmıştır. Araştırma sonuçları, birey, birey ve görev ortak etkisi ve görev ve puanlayıcı ortak etkisinin önemli bir değişkenlik kaynağı olduğunu buna karşılık görev ve puanlayıcı ana etkisinin önemsiz olduğunu göstermiştir

Keywords

• Güvenirlik
• Genellenebilirlik Kuramı
• Ölçüt Bağımlı Ölçme Araçları

References

1. Akköse, M.C. (2008). The Effectiveness of Simultaneous Prompting on Teaching Naming Kitchen Tools to Children with Developmental Disabilities: A Multiple Exemplar Approach Instruction. Unpublished master’s thesis, Anadolu University Institute of Educational Sciences, Eskişehir.
2. Akmanoğlu, N., & Batu, S. (2004). Teaching pointing to numerals to individuals with autism using simultaneous prompting. Education and Training in Developmental Disabilities, 39(4), 326-336.
3. Alharby, E.R. (2006). A comparison between two scoring methods, holistic vs. analytic using two measurement models, the Generalizability Theory and the Many-facet Rasch Measurement within the context of performance assessment. Unpublished doctoral dissertation thesis, The Pennsylvania State University, Pennsylvania.
4. Atılgan, H. (2005). Generalizability theory and a sample application for inter-reliability. The Journal of Educational Sciences and Practice, 4, 24-31.
5. Atılgan, H., & Tezbaşaran, A.A. (2005). An investigation on consistency of G and Phi coefficients obtained by generalizability theory alternative decisions study for scenarios and actual cases. Eurasian Journal of Educational Research, 18, 236- 252.
6. Baker, E., Abedi, J., Linn, R., & Niemi, D. (1996). Dimensionality and generalizability of domain-independent performance assessments. Journal of Educational Research, 89(4), 197-205.
7. Brennan, R.L. (2001). Generalizability Theory. Iowa: ACT Publications.
8. Brennan, R.L. (1992). Elements of Generalizability Theory. New York, NY: Springer- Verlog.

9. Bruckner, C.T., Yoder, P.J., & McWilliam, R.A. (2006). Generalizability and decision studies: An example using conversational language samples. Journal of Early Intervention, 28, 139-153.
10. Cronbach, J.L., Gleser, G.C., Nanda, H., & Rajaratman, N. (1972). The Dependability of Behavioral Measurements: Theory of Generalizability for Scorers and Profiles. New York, NY: John Wiley and Sons.
11. Crocker, L. and Algina, J. (1986). Introduction to Classical and Modern Test Theory. Orlando, FL: Holt, Rinehart and Winston.
12. Goodwin, L.D. & Goodwin, W.L. (1991). Research Notes: Using Generalizability Theory in Early Childhood Special Education. Journal of Early Intervention, 15, 193-204.
13. Güler, N. (2009). Generalizability Theory and Comparison of the Results of G and D Studies Computed by SPSS and GENOVA Packet Programs. Education and Science, 34, 93-103.
14. Güler, N., & Gelbal, S. (2010). Studying Reliability of Open Ended Mathematics Items According to the Classical Test Theory and Generalizability Theory. Educational Sciences: Theory & Practice, 10, 989-1019.
15. Gürsel, O. (1993). Zihinsel engelli çocukların doğal sayıları, gerçek nesneler kullanılarak eşleme resimleri işaret ederek gösterme, rakamlar gösterildiğinde söyleme becerilerinin gerçekleştirilmesinde basamaklı öğretim yöntemiyle sunulan bireyselleştirilmiş öğretim materyalinin etkililiği. Eskişehir: Anadolu Üniversitesi Yayınları.
16. Heward, W.L. (1996). Exceptional Children: An Introduction to Special Education. USA: Prentice Hall.
17. Hintze, J.M., & Matthews, W.J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral assessment. School Psychology Review, 33, 258- 270.
18. Hintze, J.M. & Petitte, H.A.P. (2001). The generalizability of CBM oral reading fluency measures across general and special education. Journal of Psychoeducational Assessment, 19(2), 158-170.
19. Hintze, J.M., Owen, S.V., Shapiro, E.S., & Daly, E.J. (2000). Generalizability of oral reading fluency measures: Application of G-theory to curriculum-based measurement. School Psychology Quarterly, 15(1), 52-68.
20. Kieffer, K. M. (1998). Why Generalizability Theory is Essential and Classical Test Theory is Often Inadequate? Paper presented at the annual meeting of the Southwestern Psychological Association, New Orleans, LA.
21. Lane, S., Liu, M., Ankenmann, R.D., & Stone, C.A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
22. Lee, G., & Fitzpatrick, A.R. (2003). The effects of a student sampling plan on estimates of the errors for students passing rates. Journal of Educational Measurement, 40(1), 17-28.
23. Lee, G., & Frisbie, D. A. (1999). Estimating Reliability Under a Generalizability Theory Model for Test Scores Composed of Testlets. Applied Measurement in Education. 12(3), 237-255.
24. Lei, P., Smith, M., & Suen, H.K. (2007). The Use of Generalizability Theory to Estimate Data Reliability in Single Subject Observational Research. Psychology in Schools, 44, 433-439.
25. Musquash, C., & O’Connor, B.P. (2006). SPSS and SAS Programs for Generalizability Theory Analysis. Behavior Research Methods, 38(3), 542-547.
26. Novak, J.R., Herman, J.L., & Gearhart, M. (1996). Establishing validity for performace- based assessments: An illustration for collections of student writing. The Journal of Educational Research, 89(4), 220-233.
27. Özkan, Ş.Y., & Gürsel, O. (2006). The Effectiveness of Simultaneous Prompting on Teaching Photo Copy Skills to Students with Mental Disabilities. Ankara Üniversitesi Eğitim Bilimleri Fakültesi Özel Eğitim Dergisi, 7, 29-45.
28. Özyürek, M. (1996). Sınıfta Davranış Yönetimi: Uygulamalı Davranış Analizi. Ankara: Karatepe Yayınları.
29. Parrott, K.A., Schuster, J.W., Collins, B.C., & Gassaway, L.J. (2000). Simultaneous prompting and instructive feedback when teaching chained tasks. Journal of Behavioral Education, 10, 3-19.
30. Reckase, M.D. (1995). The Reliability of Ratings Versus Reliability Scores. Educational Measurement: Issues and Practice, 14(4), 31.
31. Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. Newburry Park CA: Sage.
32. Sudweeks, R.R., Reeve, S., & Bradshaw, W.S. (2005). A comparison of generalizability theory and many facet measurement in analysis of college sophomore writing. Assessing Writing, 9, 236-261.
33. Tindal, G., Yovanoff, P., & Geller, J.P. (2010). Generalizability theory applied to reading assessments for students with significant cognitive disabilities. The Journal of Special Education, 44(1), 3-17.
34. Topsakal, M., & Düzkantar, A.U. (2010). The Effectiveness of Simultaneous Prompting in Teaching Car Washing to Children with Mental Retardation by Correcting Error. Abant İzzet Baysal Üniversitesi Dergisi, 10, 79-94.
35. Varol, N. (1996). Beceri Öğretimi Materyali Geliştirme ve Beceri Öğretiminde İpuçlarının Kullanımı. Gazi Üniversitesi Eğitim Fakültesi Dergisi, 16(1), 35-46.
36. Varol, N. (2004). Öz Bakım Becerilerinin Öğretimi. Ankara: Kök Yayınevi.
37. Volpe, R.J., McConaughy, S.H., & Hintze, J.M. (2009). Generalizability of Classroom Behavior Problem and On-Task Scores from the Direct Observation Form. School Psychology Review, 38, 382-401.
38. Webb, N.M., & Shavelson, R.J. (2005). Generalizability theory: Overview. B.S. Everitt & D.C. Howell (Eds.), In Encyclopedia of Statistics in Behavioral Science (pp.717-719). Hoboken, NJ: Wiley.
39. Webb, N.M., Schlackman, J., & Sugrue, B. (2000). The dependability and interchangeability of assessment methods in science. Applied Measurement in Education, 13, 277-301.