Katma-değerli değerlendirme modellerinde eşitleme önemli mi?

Öz

Bu çalışmanın amacı, eşitlenmiş ve eşitlenmemiş verilerin katma-değerli değerlendirme analizlerine etkisini incelemektir. Katma-değerli değerlendirme yaklaşımını uygulayabilmek için literatürde birçok model önerilmiştir. Bu çalışma, iki farklı katma değerli değerlendirme modeli karşılaştırmıştır: düzeltilmemiş hiyerarşik doğrusal model (UHLMM) ve genelleştirilmiş süreklilik (GP) modeli. Birinci model eşitlenmiş testler için kullanılırken, ikincisi bu varsayımı esnetir. Her iki modelde iki farklı veri seti (eşitlenmiş ve eşitlenmemiş) analiz edildi. Her iki model için eyalet çapında yapılan bir sınav (eşitlenmiş) ve ülke çapında yapılan bir sınav (eşitlenmemiş) verilerine dayanan katma-değer kestirimleri genellikle tutarlı bulundu. Okul sıralamalarında iki model arasında bazı farklılıklar gözlendi. Bu çalışmanın pratik çıkarımı, okul sıralamasında küçük farklılıklar olmasına rağmen, test formları arasında eşitlemenin mümkün olmadığı durumlarda eşitlenmemiş bir veri seti gerektiren bir modelin uygulanabileceğidir.

Anahtar Kelimeler

• Katma-değerli değerlendirme,Hiyerarşik doğrusal model,Genelleştirilmiş süreklilik modeli,Test eşitleme

Does equating matter in value-added models?

Abstract

The purpose of this study was to examine the effect of equated and non-equated data on value-added assessment analyses. Several models have been proposed in the literature to apply the value-added assessment approach. This study compared two different value-added models: the unadjusted hierarchical linear model and the generalized persistence model. The former model assumes equated tests while the latter one relaxes this assumption. Two different data sets (equated and non-equated) were analyzed with both models. Value-added estimates for both models based on a statewide examination (equated) and a countrywide examination (non-equated) data were generally consistent. School rankings showed differences between the two models. The practical implication of this study is that although there were small differences in school rankings, a model requiring an equating assumption can be applied to a non-equated data set in a case when equating between test forms is not possible.

Keywords

• Value-added assessment,Hierarchical linear model,Generalized persistence model,Test equating

References

1. Aitkin, M., & Longford, N. (1986). Statistical modeling in school effectiveness studies. Journal of the Royal Statistical Society, Series A, 149, 1–43.
2. Balcı, A. (1988). Etkili okul. Eğitim ve Bilim, 12(70), 21-30.
3. Ballou, D. (2009). Test scaling and value-added measurement. Education Finance and Policy, 4, 351–383.
4. Ballou, D., Sanders, W. L., & Wright, P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29, 37–66.
5. Beardsley, A. A. (2008). Methodological concerns about the education value-added assessment system. Educational Researcher, 37(2), 65–75.
6. Bessent, A. M., & Bessent, E. W. (1980). Determining the comparative efficiency of schools through data envelopment analysis. Educational Administration Quarterly, 16(2), 57–75.
7. Boran, A., Atalmis, E. H., Sagir, E. (2015). Özel öğretim kurs merkezi öğretmenleri ve çalışma koşulları [Private tutoring centers and their working conditions]. Turkish Journal of Education 4(4), 17–29.
8. Braun, H. (2005). Value-added modeling: What does due diligence require? In R.W. Lissitz (Ed.), Value-added models in education: Theory and application (pp. 19–40). Maple Grove, MN: JAM Press.

9. Braun, H., & Wainer, H. (2007). Value added modeling. In C.R. Rao & S. Sinharay (Eds.) Handbook of Statistics, Vol. 26. Amsterdam: Elsevier.
10. Briggs, D. C., & Domingue, B. (2013). The gains from vertical scaling. Journal of Educational and Behavioral Statistics, 38(6), 551–576.
11. Briggs, D. C., & Weeks, J. P. (2009). The sensitivity of value-added modeling to the creation of a vertical score scale. Education Finance and Policy, 4, 384–414. doi:10.1162/edfp.2009.4.4.384
12. Briggs, D. C., Weeks, J. P., & Wiley, E. (2008, April). Vertical scaling in value-added models for student learning. National Conference on Value-Added Modeling, Madison, WI.
13. Broatch, J., & Lohr, S. (2012). Multidimensional assessment of value added by teachers to real-world outcomes. Journal of Educational and Behavioral Statistics, 37, 256–277.
14. Doran, H. C., & Cohen, J. (2005). The confounding effect of linking bias on gains estimated from value-added models. In R. Lissitz (Ed.), Value-added models in education: Theory and application (pp. 80–104). Maple Grove, MN: JAM Press.
15. Ercikan, K. (2006). Development in assessment of student learning. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 929–952). Mahwah, NJ: Erlbaum.
16. Hanushek, E. A. (1972). Education and race: An analysis of the educational production process. Lexington, MA: Lexington Books.
17. Hill, H. C. (2009). Evaluating value-added models: A validity argument approach. Journal of Policy Analysis and Management, 28, 700–709. doi:10.1002/pam.20463
18. Karl, A. T., Yang, Y., & Lohr, S. (2012). GPvam: maximum likelihood estimation of multiple membership mixed models used in value-added modeling. R Package Version 2.0-0.
19. Lockwood, J. R., McCaffrey, D. F., Hamilton, L. S., Stecher, B., Le, V. N., & Martinez, J. F. (2007). The sensitivity of value‐added teacher effect estimates to different mathematics achievement measures. Journal of Educational Measurement, 44(1), 47–67.
20. Mariano, L. T., McCaffrey, D. F., & Lockwood, J. R. (2010). A model for teacher effects from longitudinal data without assuming vertical scaling. Journal of Educational and Behavioral Statistics, 35(3), 253–279.
21. Martineau, J. (2006). Distorting value-added: The use of longitudinal, vertically scaled student achievement data for value-added accountability. Journal of Educational and Behavioral Statistics, 31, 35–62.
22. Marzano, R. J. (2003). What works in schools: Translating research into action? Alexandria, VA: ASCD.
23. McCaffrey, D., Lockwood, J. R., Koretz, D., & Hamilton, L. (2003). Evaluating value-added models for teacher accountability. Washington, DC: RAND.
24. McCaffrey, D., Lockwood, J., Koretz, D., Louis, T., & Hamilton, L. (2004). Models for value added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67–101.
25. Murnane, R. J. (1975). The impact of school resources on the learning of children. Cambridge, MA: Ballinger Publishing.
26. Raudenbush, S. & Bryk, A. S. (1986). A hierarchical model for studying school effects. Sociology of Education, 59, 1–17.
27. Reckase, M. D. (2004). The real world is more complicated than we would like. Journal of Educational and Behavioral Statistics, 29, 117–120.
28. Sanders, W. L. & Horn, S. P. (1994). The Tennessee Value-Added Assessment System (TVAAS): Mixed model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8, 299–311.
29. Sanders, W. L., & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville: University of Tennessee, Value-Added Research and Assessment Center.
30. Sanders, W. L., Saxton, A. M., & Horn, S. P. (1997). The Tennessee Value-Added Assessment System: A quantitative, outcomes-based approach to educational assessment. In J. Millman, (Ed.), Grading teachers, grading schools. Is student achievement a valid evaluation measure? (pp. 137–162). Thousand Oaks, CA:Corwin.
31. Sanders, W. L., Saxton, A., Schneider, J., Dearden, B., Wright, S. P., & Horn, S. (2002). Effects of building change on indicators of student achievement growth: Tennessee Value-Added Assessment System. Knoxville, TN: University of Tennessee Value-Added Research and Assessment Center.
32. Shaw, L. H. (2012). Incorporating latent variable outcomes in value-added assessment: An evaluation of univariate and multivariate measurement model structures. (Unpublished doctoral dissertation). University of Nebraska, Digital Commons at the University of Nebraska-Lincoln.
33. Şen, S., Kim, S.-H., & Cohen, A. S. (2017). Comparative analysis of common statistical models used for value-added assessment of school performance. Journal of Measurement and Evaluation in Education and Psychology, 8(3), 303–320.
34. Tekwe, C. D., Carter, R. L., Ma, C., Algina, J., Lucas, M. E., Roth, J., … Resnick, M. B. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29, 11–36.
35. Wright, S. P., Horn, S. P., & Sanders, W. L. (1997). Teacher and classroom context effects on student achievement: Implications for teacher evaluation. Journal of Personnel Evaluation in Education, 1(1), 57–67.
36. Yıldırım, İ., & Şen, S. (2018). Katma-değerli değerlendirme modellerinde test eşitleme durumunun incelenmesi. In S. Dinçer, (Ed.), Değişen dünyada eğitim (pp. 125–134). Ankara: Pegem Akademi.