Examination of Student Growth Using Gain Score and Categorical Growth Models

Abstract

In this study, gain score, and categorical growth models were used to examine the role of student (gender and socioeconomic level) and school characteristics (school size and school resources) in the student growth on comprehension skills in language. The participants of this study were 2,416 sixth-grade students in 2011 who became seventh-grade students in 2012. The data was collected through two achievement tests, student and school questionnaires. Two achievement tests were calibrated using the Rasch Model and were scaled using the concurrent estimation method. Moreover, the cut-off scores of these tests were determined by using the bookmark method. Students’ growth was modelled with the gain score and categorical growth models. All data was analyzed using multilevel models. Results showed that some students did not achieve sufficient gains to advance to higher performance levels. Although some schools’ average gains were higher, their performance was still not significant enough in terms of tests’ standards. Moreover, the analyses demonstrated that the student gain scores and growth categories varied significantly among the schools. In addition, the study was able to determine student and school characteristics that have an impact on the students' gain scores and categorical growth. Given the different aspects gained about students’ performance with these models, it is recommended to utilize different growth models in schools. 

Keywords

• Gain score model
• Categorical growth model
• Student growth
• Multilevel modeling

Examination of Student Growth Using Gain Score and Categorical Growth Models

Abstract

In this study, gain score, and categorical growth models were used to examine the role of student (gender and socioeconomic level) and school characteristics (school size and school resources) in the student growth on comprehension skills in language. The participants of this study were 2,416 sixth-grade students in 2011 who became seventh-grade students in 2012. The data was collected through two achievement tests, student and school questionnaires. Two achievement tests were calibrated using the Rasch Model and were scaled using the concurrent estimation method. Moreover, the cut-off scores of these tests were determined by using the bookmark method. Students’ growth was modelled with the gain score and categorical growth models. All data was analyzed using multilevel models. Results showed that some students did not achieve sufficient gains to advance to higher performance levels. Although some schools’ average gains were higher, their performance was still not significant enough in terms of tests’ standards. Moreover, the analyses demonstrated that the student gain scores and growth categories varied significantly among the schools. In addition, the study was able to determine student and school characteristics that have an impact on the students' gain scores and categorical growth. Given the different aspects gained about students’ performance with these models, it is recommended to utilize different growth models in schools. 

Keywords

• Gain score model
• Categorical growth model
• Student growth
• Multilevel modeling

References

1. Anıl, D., Özer Özkan, Y. ve Demir, E. (2015). PISA 2012 araştırması ulusal nihai rapor. T.C. Millî Eğitim Bakanlığı, Ölçme, Değerlendirme ve Sınav Hizmetleri Genel Müdürlüğü, Ankara.
2. Arnold, D. H., & Doctoroff, G. L. (2003). The early education of socioeconomically disadvantaged children. Annual Review of Psychology, 54, 517–545.
3. Ayral, M., Özdemir, N. ve Sadıç, Ş. (2011). Altındağ İlçesi Öğrenme Düzeyi Araştırması Raporu. Altındağ Rehberlik ve Araştırma Merkezi, Ankara. Retrieved February 20, 2017 from http://altindagram.meb.k12.tr/meb_iys_dosyalar/06/01/334395/dosyalar /2012_12/18012758_renmedzeyiaratrmas_1.pdf adresinden erişildi.
4. Betebenner, D. W. (2009). Growth, standards and accountability. Retrieved February 25, 2017 from the National Center for the Improvement of Educational Assessment: http://www.nciea.org/publications/normative_criterion_growth_DB08.pdf
5. Betebenner, D. W. (2011). SGP: An R package for the calculation and visualization of student growth percentiles & percentile growth trajectories [R package version 0.7–1.0].
6. Briggs, D., & Betebenner, D. W. (2009, April). Is growth in student achievement scale dependent? Paper presented at the invited symposium Measuring and Evaluating. Changes in Student Achievement: A Conversation About Technical and Conceptual Issues at the annual meeting of the National Council for Measurement in Education, San Diego, CA.
7. Browne, W. J. & Draper, D. (2006). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1, 473–514.
8. Bursal, M., Buldur, S. & Dede, Y. (2015). Alt sosyo-ekonomik düzeyli ilköğretim öğrencilerinin 4-8. sınıflar fen ve matematik ders başarıları: Cinsiyet perspektifi [Science and mathematics course success of elementary students in low socio-economic status among 4th-8th grades: Gender perspective]. Eğitim ve Bilim, 40(179) 133-145.

9. Büyüköztürk, Ş., Çakan, M., Tan, Ş. ve Atar, H. Y. (2014). TIMSS 2011 ulusal matematik ve fen raporu- 8. Sınıflar. T.C. Millî Eğitim Bakanlığı, Ölçme, Değerlendirme ve Sınav Hizmetleri Genel Müdürlüğü, Ankara.
10. Castellano, K. E., & Ho, A. D. (2013). A practitioner’s guide to growth models. Washington, DC: Council of Chief State School Officers. Retrieved February 20, 2017 from: http:// scholar.harvard.edu/files/andrewho/filesa_pracitioners_guide_to_growth_models.pdf
11. Chavent, M., Kuentz-Simonet, V., Labenne, A., & Saracco, J. (2014). Multivariate analysis of mixed data: The PCAmixdata R package. arXiv preprint arXiv:1411.4911.
12. Cheti, N., & Birgitta, R. (2012). The effect of school resources ontest scores in England, ISER Working Paper Series, No. 2012-13.
13. Crawford, L., Tindal, G., & Steiber, S. (2001). Using oral reading rate to predict student performance on statewide achievement tests. Educational Assessment, 7(4), 303–323.
14. Denton, K., & West, J. (2002). Children’s reading and mathematics achievement in kindergarten and first grade (NCES 2002-125). Washington, DC: National Center for Education Statistics.
15. Dinçer, M. A. ve Oral, I. (2010). Türkiye’de devlet liselerinde akademik yılmazlık profili: PISA 2009 Türkiye verisinin analizi. İstanbul: Eğitim Reformu Girişimi.
16. Ergin-Aydemir, S. ve Sünbül, Ö. (2016). Matematik bilişsel gelişiminin örtük büyüme modeli ile izlenmesi [Monitoring the mathematics cognitive development with the latent growth modeling]. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 16(1), 20-40.
17. Erman-Aslanoğlu, A. ve Kutlu, Ö. (2015). Factors related to the reading comprehension skills of 4th grade students according to data of PIRLS 2001 Turkey. Journal of Educational Sciences Research, 5(2), 1-18.
18. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect?. School Psychology Review, 22, 27-27.
19. Glewwe, P. W., Hanushek, E. A., Humpage, S. D., & Ravina, R. (2011). School resources andeducational outcomes in developing countries: a review of the literature from 1990 to 2010.Working Paper 17554. Cambridge, MA: National Bureau of Economic Research.
20. Goldschmidt, P., Choi, K., & Beaudoin, J. P. (2012). Growth model comparison study: Practical implications of alternative models for evaluating school performance. Washington, DC: Council of Chief State School Officers.
21. Gong, B. (2004). Models for using student growth measures in school accountability. Paper presented at the Council of Chief State School Officers’ “Brain Trust” on Value-added Models, Washington, DC.
22. Gonzalez, J. & Wilberg, M. (2017). Applying Test Equating Methods Using R. New York: Springer International Publishing.
23. Güzle-Kayır, Ç. ve Erdoğan, M. (2015). The variation in Turkish students’ reading skills based on PISA 2009: The effects of socio-economic and classroom-related factors. International Online Journal of Educational Sciences, 7(4), 80 – 96.
24. Hanson, B. A., & Beguin, A. A. (2002). Obtaining a common scale for the item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26, 3–24.
25. Hanushek, E. A. (2006). School resources. in Hanushek, E. A., and Welch, F. (eds) Handbook of the Economics of Education, (Amsterdam: Elsevier) 865-908
26. Heck, R. H. (2006). Assessing school achievement progress: Comparing alternative approaches. Educational Administration Quarterly, 42(5), 667-699.
27. Herbers, J. E., Cutuli, J. J., Supkoff, L. M., Heistad, D., Chan, C.-K., Hinz, E., & Masten, A. S. (2012). Early reading skills and academic achievement trajectories of students facing poverty, homelessness, and high residential mobility. Educational Researcher, 41, 366–374.
28. Hughes, J. N., Luo, W., Kwok, O.-M., & Loyd, L. K. (2008). Teacher-student support, effortful engagement, and achievement: A 3-year longitudinal study. Journal of Educational Psychology, 100(1), 1–14.
29. Husain, M., & Millimet, D. (2009). The mythical ‘‘boy crisis’’? Economics of Education Review, 28, 38-48.
30. Krueger, A. B., & Lindahl (2001). Education for growth: Why and for whom?, Journal of Economic Literature, 39(4), 1101–1136.
31. Kurdek, L. A., & Sinclair, R. J. (2001). Predicting reading and mathematics achievement in fourth-grade children from kindergarten readiness scores. Journal of Educational Psychology, 93(3), 451−455.
32. Kutlu, Ö., Yıldırım, Ö., Bilican, S. ve Kumandaş, H. (2011). İlköğretim 5. sınıf öğrencilerinin okuduğunu anlamada başarılı olup olmama durumlarının kestirilmesinde etkili olan değişkenlerin incelenmesi. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 2(1), 132-139.
33. Laird, E. (2008). Tapping into the power of longitudinal data: A guide for school leaders. Retrieved January 21, 2018 from http://www.dataqualitycampaign.org/.
34. Leithwood, K., Edge, K., & Jantzi, D. (1999). Educational accountability: The state of the art. Gütersloh, Germany: Bertelsmann Foundation.
35. Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996, June). Standard setting: A Bookmark approach. In D. R. Green (Chair), IRT-based standard setting procedures utilizing behavioral anchoring. Symposium conducted at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix AZ.
36. Liu, Y., Zumbo, B. D., & Wu, A. D. (2014). Relative importance of predictors in multilevel modeling. Journal of Modern Applied Statistical Methods, 13(1), 1-22.Luppescu, S. (2005). Virtual equating. Rasch Measurement Transactions, 19(3), 10-25.
37. McCoach, B. D., O’Connell, A. A., Reis, S. M., & Levitt, H. A. (2006). Growing readers: A hierarchical linear model of children’s reading growth during the first 2 years of school. Journal of Educational Psychology, 98, 14-28.
38. Morgan, P. L., Farkas, G., & Wu, Q. (2011). Kindergarten children’s growth trajectories in reading and mathematics: Who falls increasingly behind? Journal of Learning Disabilities, 44, 472- 488.
39. Mullis, I. V. S., Martin, M. O., Gonzalez, E. J., & Kennedy, A. M. (2001). PIRLS 2001 International Report. Chestnut Hill, MA: International Study Center. Retrieved February 20, 2017 from https://timssandpirls.bc.edu/pirls2001i/pdf/p1_IR_book.pdf.
40. Muthén, L. K., & Muthén, B. O. (1998-2017). Mplus User’s Guide (Eighth Edition). Los Angeles, CA: Muthén & MuthénNational Assessment Agency. (2008). 14-19 Reforms. Retrieved February 3, 2017 from: http://www.naa.org.uk/
41. Nayır, F. (2013). Eğitimde kalite geliştirme sürecinde okul değerlendirmenin rolü [The role of school evaluation in the process of improving quality of education]. Celal Bayar Üniversitesi Sosyal BilimlerDergisi, 11(2), 119-134.
42. Nese, J., Biancarosa, G., Anderson, D., Lai, C., Alonzo, J., & Tindal, G. (2012). Within- year oral reading fluency with CBM: A comparison of models. Reading & Writing, 25(4), 887-915.
43. No Child Left Behind Act of 2001, P. L. 107-110, 20 U.S.C. 6319 (2002).
44. OECD (2014a). PISA 2012 Results: What Students Know and Can Do – Student Performance in Mathematics, Reading and Science (Volume I, Revised edition, February 2014), PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264201118-en
45. OECD. (2014b). PISA 2012 technical report. Paris: OECD Publications.
46. OECD (2016). PISA 2015 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264266490-en
47. Osborne, J. W. (2000). Advantages of hierarchical linear modeling. Practical Assessment, Research, and Evaluation, 7(1), 1-3.
48. Özer Özkan, Y. (2016). Okulları başarılarına göre sınıflandırmada etkili olan değişkenlerin PISA 2012 Türkiye verileri aracılığıyla incelenmesi [The impact of school properties to mathematics literacy in the PISA 2012 Turkey sample]. International Online Journal of Educational Sciences, 8(2), 117-130.
49. Palardy, G. J. (2008). Differential school effects among low, middle, and high social class composition schools: A multiple group, multilevel latent growth curve analysis. School Effectiveness and School Improvement, 19, 21–49.
50. Pike, G. R. (1991). Using structural equation models with latent variables to study student growth and development. Research in Higher Education, 32(5), 499-524.
51. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publision.
52. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., & Congdon, R. T., Jr. (2011). HLM 7: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.
53. Rock, D. A., & Pollack, J. M. (2002). A model‐based approach to measuring cognitive growth in pre‐reading and reading skills during the kindergarten year. ETS Research Report Series, 2002(2).
54. Ryser, G., & Rambo-Hernandez, K. (2014). Using growth models to measure school performance: Implications for gifted learners. Gifted Child Today, 37(1), 17–23.
55. Salvucci, S., Walter, E., Conley, V., Fink, S., & Saba, M. (1997). Measurement error studies at the National Center for Education Statistics (NCES). Washington D. C.: U. S. Department of Education.
56. Shin, T., Davison, M. L., Long, J. D., Chan, C., & Heistad, D. (2013). Exploring gains in reading and mathematics achievement among regular and exceptional students using growth curve modeling. Learning and Individual Differences, 23, 92-100.
57. Skibbe, L. E., Connor, C. M., Morrison, F. J., & Jewkes, A. M. (2010). Schooling effects on preschoolers’ self-regulation, early literacy, and language growth. Early Childhood Research Quarterly, 26, 42-49.
58. Slaughter, R. (2008). Measuring middle school achievement growth with student growth percentile methodology. NERA Conference http://digitalcommons.uconn.edu/nera_2008/20
59. Stevenson, H. (2006) Moving towards, into and through principalship: Developing a framework for researching the career trajectories of school leaders. Journal of Educational Administration, 44(4), 408-420.
60. Taş, U. E., Arıcı, Ö., Özarkan, H. B. ve Özgürlük, B. (2016). PISA 2015 ulusal raporu. T.C. Millî Eğitim Bakanlığı, Ölçme, Değerlendirme ve Sınav Hizmetleri Genel Müdürlüğü, Ankara.
61. Türkoğlu, M. E. ve Aypay, A. (2015). Özel okul öğretmenlerinin öğretmen hesap verebilirliğine dair düşünceleri [Private school teachers’ thoughts about teacher accountability]. Eğitimde Politika Analizi Dergisi, 4(1), 7-32.
62. Twisk, J., & de Vente, W. (2002). Attrition in longitudinal studies: How to deal with missing data. Journal of Clinical Epidemiology, 55, 329-37.
63. U.S. Department of Education. (2010, June 24). Race to the Top Assessment Program. Announcement of Race to the Top Program by U.S. Department of Education. Retrieved February 18, 2017 from http://www2.ed.gov/programs/racetothetop -assessment/index.html
64. Welch, C., Dunbar, S., & Rickels, H. (2016). Measuring student growth in Iowa with Iowa assessments. Retrieved February 25, 2017 from https://itp.education.uiowa.edu/ia/documents/Measuring-Student-Growth-in-Iowa-with-the-Iowa-Assessments.pdf
65. Yapar, T. (2014). İngilizce okuma becerisindeki gelişimin Madde Tepki Kuramı ve örtük büyüme modellemesiyle incelenmesi [An investigation of the development in English reading skill by using item response theory and latent growth modeling]. (Unpublished doctorate thesis). Hacettepe Üniversitesi, Ankara.
66. Yavuz, H. Ç., & Kutlu, Ö. (2016). Ekonomik Bakımdan Dezavantajlı Öğrencilerin Akademik Yılmazlık Düzeylerinin Bazı Koruyucu Faktörler Açısından İncelenmesi [Investigation of the factors affecting the academic resilience of economically disadvantaged high school students]. Eğitim ve Bilim, 41(186), 1-19.