Thematic Content Analysis of Studies Using Generalizability Theory

Abstract

One of the important theories in education and psychology is Generalizability (G) Theory and various properties distinguish it from the other measurement theories. To better understand methodological trends of G theory, a thematic content analysis was conducted. This study analyzes the studies using generalizability theory in the field of education in Turkey by using the method of thematic content analysis. It reviews 60 studies, including 31 articles and 29 theses published from 2004 to 2017. The selected studies underwent thematic content analysis using parameters including tagged information, aim, G Theory type, number of facets used in the study, Turkish word for “facet,” object of measurement, sample size, design type, mixed-design availability, shared results of G and D studies, computer programs, method of calculating negative variance, availability of fixed facets, and design balance. The data were interpreted on the basis of frequencies; both table and figures are included in the study. According to the results, there is an increase in the number of studies conducted by using G theory by years. Of these, many compare theories; most of them applying univariate G Theory and consider two-faceted measurement situations. While a small subset of studies features mixed design, a large group features crossed design, with individuals as the object of measurement. The computer program most commonly used in analyses is EduG. The majority of studies use balanced design. Recommendations are provided accordingly with the results.

Keywords

• Educational sciences
• Generalizability theory
• Thematic content analysis

Thematic Content Analysis of Studies Using Generalizability Theory

Abstract

One of the important theories in education and psychology is Generalizability (G) Theory and various properties distinguish it from the other measurement theories. To better understand methodological trends of G theory, a thematic content analysis was conducted. This study analyzes the studies using generalizability theory in the field of education in Turkey by using the method of thematic content analysis. It reviews 60 studies, including 31 articles and 29 theses published from 2004 to 2017. The selected studies underwent thematic content analysis using parameters including tagged information, aim, G Theory type, number of facets used in the study, Turkish word for “facet,” object of measurement, sample size, design type, mixed-design availability, shared results of G and D studies, computer programs, method of calculating negative variance, availability of fixed facets, and design balance. The data were interpreted on the basis of frequencies; both table and figures are included in the study. According to the results, there is an increase in the number of studies conducted by using G theory by years. Of these, many compare theories; most of them applying univariate G Theory and consider two-faceted measurement situations. While a small subset of studies features mixed design, a large group features crossed design, with individuals as the object of measurement. The computer program most commonly used in analyses is EduG. The majority of studies use balanced design. Recommendations are provided accordingly with the results.

Keywords

• Educational sciences
• Generalizability theory
• Thematic content analysis

References

1. Arık, R. S. & Türkmen, M. (2009). Examination of the articles in the scientific journals published in the field of educational sciences. Paper presented at I. International Congress of Educational Research, Çanakkale, Turkey.
2. Atılgan, H. (2004). Genellenebilirlik kuramı ve çok değişkenlik kaynaklı Rasch modelinin karşılaştırılmasına ilişkin bir araştırma [A research on the comparison of the generalizability theory and many facet Rasch model] (Doctoral Dissertation). Hacettepe University, Ankara.
3. Atılgan, H. (2013). Sample size estimation of G and Phi coefficients in generalizability theory. Eurasian Journal of Educational Research, 51, 215–228.
4. Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher, 36, 258–267.
5. Bektaş, M., Dündar, H. & Ceylan, A. (2013). Investigation of several variables papers national classroom teacher education symposium. Uşak University Journal of Social Sciences, 6(2), 201–226. DOI: http://dx.doi.org/10.12780/UUSBD167
6. Bloch, R. & Norman, G. (2011). G String 4 user manual (Version 6.1.1). Hamilton, Ontario, Canada. Retrieved from http://fhsperd.mcmaster.ca/g_string/download/g_string_4_manual_ 611.pdf
7. Brennan, R. L. (2011). Generalizability Theory and Classical Test Theory. Applied Measurement in Education, 24, 1–21. doi:10.1080/08957347.532417.
8. Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.

9. Brennan, R. L. (1997). A perspective on the history of Generalizability Theory. Educational Measurement: Issues and Practice, 16(4), 14–20. https://doi.org/10.1111/j.1745-3992.1997.tb00604.x
10. Brennan, R. L. (1992). Elements of Generalizability Theory. NY: Springer-Verlag.
11. Çalık, M. & Sözbilir, M. (2014). Parameters of content analysis. Education and Science, 39(174), 33–38. doi:10.15390/EB.2014.3412
12. Çiltaş, A. (2012). Content analysis of the graduate thesis and dissertations in mathematics education in Turkey between 2005-2010. The Journal of Academic Social Science Studies, 5(7), 211–228.
13. Çiltaş, A., Güler, G. & Sözbilir, M. (2012). Mathematics education research in Turkey: A content analysis study. Educational Sciences: Theory & Practice, 12(1), 565–580.
14. Deliceoğlu, G. (2009). The Comparison of the reliabilities of the soccer abilities’rating scale based on the Classical Test Theory and Generalizability. (Doctoral Dissertation). Ankara University, Ankara.
15. Doğru, M., Gençosman, T., Ataalkın, A. N. & Şeker, F. (2012). Fen bilimleri eğitiminde çalışılan yüksek lisans ve doktora tezlerinin analizi [Analysis of master's and doctoral theses in science education]. Journal of Turkish Science Education, 9(1), 49–64.
16. Finfgeld, D. L. (2003). Metasynthesis: The state of the art-so far. Qualitative Health Research, 13(7), 893–904. DOI: 10.1177/1049732303253462
17. Gadbury-Amyot, C. C., Kim, J., Palm, R. L., Mills, G. E., Noble, E. & Overman, P. R. (2003). Validity and reliability of portfolio assessment of competency in a baccalaureate dental hygiene program. Journal of Dental Education, 67(9), 991-1002.
18. Göktaş, Y., Küçük, S., Aydemir, M., Telli, E., Arpacık, Ö., Yıldırım, G. & Reisoğlu, İ. (2012). Educational technology research trends in Turkey: A content analysis of the 2000–2009 decade. Educational Sciences: Theory & Practice, 12(1), 191–196.
19. Gülbahar, Y. & Alper, A. (2009). Trends and issues in educational technologies: A review of recent research in TOJET. The Turkish Online Journal of Educational Technology – TOJET, 8(2), 124-135.
20. Güler, N., Kaya Uyanık, G. & Taşdelen Teker, G. (2012). Genellenebilirlik Kuramı [Generalizability Theory]. Ankara: PegemA Yayıncılık.
21. Güler, N. (2008). A research on classical test theory, generalizability theory and Rasch model (Doctoral Dissertation). Hacettepe University, Ankara.
22. Günay, R. & Aydın, H. (2015). Inclinations in studies into multicultural education in Turkey: A content analysis study. Education and Science, 40(178), 1–22. DOI: http://dx.doi.org/10.15390/EB.2015.3294
23. Hambleton, R. K. & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 3847. http://dx.doi.org/10.1111/j.1745-3992.1993.tb00543.x
24. Kaleli Yılmaz, G. (2015). Analysis of technological pedagogical content knowledge studies in Turkey: A meta-synthesis study. Education and Science, 40(178), 103–122. DOI: http://dx.doi.org/10.15390/EB.2015.4087
25. Karadağ, E. (2009). Eğitim bilimleri alanında yapılmış doktora tezlerinin incelenmesi [A Thematic Analysis on Doctoral Dissertations Made In the Area of Education Sciences],. Ahi Evran Üniversitesi Eğitim Fakültesi Dergisi 10(3), 75–87.
26. Kersting, N. (2008). Using Video Clips of Mathematics Classroom Instruction as Item Prompts to Measure Teachers’ Knowledge of Teaching Mathematics. Educational and Psychological Measurement, 68(5), 845-861. DOI:10.1177/0013164407313369
27. Kılıç Çakmak, E., Çebi, A., Mihçi, P., Günbatar, M. S. & Akçayir, M. (2013). A content analysis of educational technology research in 2011. 4th International Conference on New Horizons in Education. INTE 2013 Proceedings Book, 397–409.
28. Lafave, M. R. & Butterwick, D. J. (2014). A generalizability theory study of athletic taping using the Technical Skill Assessment Instrument. Journal of Athletic Training, 49(3), 368-372. doi: 10.4085/1062-6050-49.2.22
29. Lane, S., Liu, M., Ankenmann, R. D. & Stone, C. A. (1996). Generalizability and Validity of a Mathematics Performance Assessment. Journal of Educational Measurement, 33(1), 71-92. https://doi.org/10.1111/j.1745-3984.1996.tb00480.x
30. Miles, M, B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded Sourcebook. (2nd ed). Thousand Oaks, CA: Sage.
31. Patton, M.Q. (2002). Qualitative research and evaluation methods (3rd Ed.). London: Sage Publications, Inc.
32. Pekin Z., Çetin S. & Güler N. (2018). Comparison of Interrater Reliability Based on Different Theories for Autism Social Skills Profile. Journal of Measurement and Evaluation in Education and Psychology, 9(2), 202-215. https://doi.org/10.21031/epod.388590
33. Rios, J.A., Li, X., & Faulkner-Bond, M. (2012, October). A review of methodological trends in generalizability theory. Paper presented at the annual conference of the Northeastern Educational Research Association, Rocky Hill, CT.
34. Saban, A. (2009). Content analysis of Turkish studies about the multiple intelligences theory. Educational Sciences: Theory & Practice, 9(2), 833–876.
35. Shavelson, R. J. & Webb, N. M. (1991). Generalizability Theory: A primer. Newbury Park, CA: Sage.
36. Shavelson, R. J., Baxter, G. P. & Pine, J. (1991). Performance Assessment in Science. Applied Measurement in Education, 4(4), 347-362. DOI: 10.1207/s15324818ame0404_7
37. Taşdelen Teker, G. & Odabaşı, O. (2018). Reliability of scores obtained from standardized patient and instructor assessments. European Journal of Dental Education, 23, 88-94. DOI: 10.1111/eje.12406
38. Turner, A. A., Lozano-Nieto, A. & Bouffard, M. (2006). Generalizability of extracellular-to-intracellular fluid ratio using bio-impedance spectroscopy. Physiological Measurement, 27(4), 385-397. DOI: 10.1088/0967-3334/27/4/005
39. Yalçın, S., Yavuz, H. Ç. & İlgün Dibek, M. (2015). Content analysis of papers published in educational journals with high ımpact factors. Education and Science, 40 (182), 1–28. DOI:10.15390/EB.2015.4868
40. Yin, Y. & Shavelson, R. J. (2008). Application of Generalizability Theory to Concept Map Assessment Research. Applied Measurement in Education, 21(3), 273-291. DOI: 10.1080/08957340802161840
41. Walsh, D. & Downe, S. (2005). Meta-synthesis method for qualitative research: A literature review. Journal of Advanced Nursing, 50(2), 204–211.
42. Webb, N. M., Rowley, G. L., & Shavelson, R. J. (1988). Using Generalizability Theory in counseling and development. Measurement and Evaluation in Counseling and Development, 21, 81–90.