RELIABILITY OF INTERVIEWS USED FOR ADMISSION TO TEACHER EDUCATION PROGRAMS IN NORTHERN CYPRUS: GENERALIZABILITY THEORY

Abstract

This study examined the reliability of raters' assessments of prospective teachers for admission to a teacher training college. A total of fifty-eight prospective teachers and 10 raters participated in the study. The raters gave their ratings according to five dimensions, namely general culture (GC), language (L), self-image (SI), hobbies (H) and attitude towards the teaching profession (ATTP) in two interviews (Interview 1 and Interview 2). The data were analysed using the Mann-Whitney U-test and Generalizability Theory. The results of the study show that there is no significant difference between the raters' ratings in each interview. The G coefficients were unacceptable for the GC and SI dimensions in Interview 1. In addition, the G coefficients were relatively low for the L, H, and ATTP dimensions in Interview 1 and all dimensions in Interview 2.

Keywords

• Admission to teacher education programs
• interview
• reliability
• generalizability theory

References

1. Andrew, M. D., Cobb, C. D., & Giampietro, P. J. (2005). Verbal ability and teacher effectiveness. Journal of teacher education, 56(4), 343-354. https://doi.org/10.1177/0022487105279928
2. Arnon, S. and Reichel, N. (2007). Who is the ideal teacher? Am I? Similarity and differences in perception of students of education regarding the qualities of a good teacher and their qualities as teachers. Teachers and Teaching: Theory and Practice, 13 (5), 441-446. https://doi.org/10.1080/13540600701561653
3. Atılgan, H. (2008). Using generalizability theory to assess the score reliability of the special ability selection examinations for music education programmes in higher education. International Journal of Research & Method in Education, 31(1), 63-76. https://doi.org/10.1080/17437270801919925
4. Atılgan, H. & Tezbaşaran, A. A. (2005). An Investigation on consistency of g and phi coefficients obtained by generalizability theory alternative decisions study for scenarios and actual cases. Eurasian Journal of Educational Research, 18, 28-40.
5. Bardach, L. & Klassen, R. M. (2020). Smart teachers, successful students? A systematic review of the literature on teachers’ cognitive abilities and teacher effectiveness. Educational Research Review, 30, 100312. Advance Online Publication. https://doi.org/10.1016/j.edurev.2020.100312
6. Brennan, R. (1983). Elements of generalizability theory. Iowa City, IA: American College Testing.
7. Brennan, R. L. (2001) Generalizability theory. New York: Springer-Verlag.
8. Briesch, A. M., Swaminathan, H., Welsh, M., & Chafouleas, S. M. (2014). Generalizability theory: A practical guide to study design, implementation, and interpretation. Journal of School Psychology, 52(1), 13-35. https://doi.org/10.1016/j.jsp.2013.11.008

9. Byrnes, D., Kiger, G., & Shechtman, Z. (2003). Evaluating the use of group interviews to select students for teacher-education programs. Journal of Teacher Education, 54(2), 163-172. https://doi.org/10.1177/0022487102250310
10. Cardinet, J., Johnson, S. & Pini, G. (2011). Applying generalizability theory using EduG, New York: Taylor and Francis.
11. Casey, C., & Childs, R. (2007). Teacher education program admission criteria and what beginning teachers need to know to be successful teachers. Canadian Journal of Educational Administration and Policy, (67).
12. Caskey, M. M., Peterson, K. D., & Temple, J. B. (2001). Complex admission selection procedures for a graduate preservice teacher education program. Teacher Education Quarterly, 28(4), 7-21.
13. Corcoran, R. P., & O’Flaherty, J. (2018). Factors that predict pre-service teachers’ teaching performance. Journal of Education for Teaching, 44(2), 175-193. https://doi.org/10.1080/02607476.2018.1433463
14. Crocker, L. M., & Algina, L. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart and Winston.
15. Cronbach, L.J., Rajarathnam, N. & Gleser, G.C. (1963). Theory of generalizability: a liberalization of reliability theory. British Journal of Statistical Psychology, 16(2), 137–163.
16. Donnon, T., & Paolucci, E. O. (2008). A generalizability study of the medical judgment vignettes interview to assess students' noncognitive attributes for medical school. BMC Medical Education, 8(1), 1-7. https://doi.org/10.1186/1472-6920-8-58
17. Ebmeier, H., & Ng, J. (2005). Development and field test of an employment selection instrument for teachers in urban school districts. Journal of Personnel Evaluation in Education, 18(3), 201e218. https://doi.org/10.1007/s11092-006-9021-4
18. Faulk, L. G. (2008). Predicting on-the-job teacher success based on a group assessment procedure used for admission to teacher education (Doctoral dissertation). Available from ProQuest Dissertations and Thesis database (UMI No. 3297518).
19. Furr, M. (2011). Scale construction and psychometrics for social and personality psychology, London: Sage.
20. Goho, J. & Blackman, A. (2006). The effectiveness of academic admission interviews: an exploratory meta-analysis. Medical Teacher, 28(4), 335-340. https://doi.org/10.1080/01421590600603418
21. Güler, N., Uyanık, G. K. & Teker, G. T. (2012). Generalizability theory. Ankara: Pegem Academic Publishing.
22. Gürlen, E., Boztunç-Öztürk, N., & Eminoğlu, E. (2019). Investigation of the reliability of teachers, self and peer assessments at the primary school level with generalizability theory. Journal of Measurement and Evaluation in Education and Psychology, 10(4), 406-421. https://doi.org/10.21031/epod.583891
23. Haberman, M. & Post, L. (1998).Teachers for multicultural schools: the power of selection. Theory into Practice, 37(2), 96-104. https://doi.org/10.1080/00405849809543792
24. Huffcutt, A., & Woehr, D. (1999). Further analysis of employment interview validity: A quantitative evaluation of interviewer-related structuring methods. Journal of Organizational Behavior, 20, 549–560.
25. Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
26. Kim, L. E., Jörg, V., & Klassen, R. M. (2019). A meta-analysis of the effects of teacher personality on teacher effectiveness and burnout. Educational psychology review, 31, 163-195. https://doi.org/10.1007/s10648-018-9458-2
27. Leger, K. E. (2014). Defining teaching excellence: A phenomenological study of 2013 highly effective louisiana value-added model teachers with perfect evaluation scores. (Unpublished doctoral dissertation). Faculty of the College of Graduate Studies, Lamar University.
28. Lupascu, A. R., Pânisoarâ, G., & Pânisoarâ, I-O. (2014). Characteristics of effective teacher. Procedia. Social and Behavioral Sciences, 127, 534-538. https://doi.org/10.1016/j.sbspro.2014.03.305
29. Marcoulides, G. A. (1993). Maximizing Power in Generalizability Studies under Budget Constraints, Journal of Educational Statistics, 18(2), 197-206.
30. Mathis, R. L., Jackson, J. H., Valentine, S. R., & Meglich, P. (2016). Human resource management. Cengage Learning.
31. McDaniel, M. A., Whetzel, D. J., Schmidt, F. T., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599–616. https://doi.org/10.1037/0021-9010.79.4.599
32. Petrarca, D., & LeSage, A. (2014). Should it stay or should it go? Re-considering the Pre-service Teacher Education Admissions Interview. ICET 2014, 252.
33. Pellicer, L. O. (1981). Improved teacher selection with the structured interview. Educational Leadership, 38(6), 492-94.
34. Pursell, E. D., Campion, M. A., & Gaylord, S. R. (1980). Structured interviewing: Avoiding selection problems. Personnel Journal, 59, 907–912.
35. Schulte, D. P., Slate, J. R., & Onwuegbuzie, A. J. (2008). Effective high school teachers: A mixed investigation. International Journal of Educational Research, 47(1), 351-361. https://10.1016/j.ijer.2008.12.001
36. Shavelson, R. J. & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage
37. Shechtman, Z. (1992). Interrater reliability of a single group assessment procedure administered in several educational settings. Journal of Personnel Evaluation in Education, 6(1), 31-39.
38. Smith, H. A., & Pratt, D. (1996). The use of biodata in admissions to teacher education. Journal of Teacher Education, 47(1), 43-52. https://doi.org/10.1177/0022487196047001008
39. Stronge, J. H. (2007). Qualities of effective teachers. Alexandria, VA: Association for Supervision and Curriculum Development.
40. Taneri, P. O. (2017). The viewpoints of instructors about the effects of teacher education programs on prospective teachers’ affective characteristics. Eurasian Journal of Educational Research, 17 (70), 105-120. http://dx.doi.org/10.14689/ejer.2017.70.6
41. Taşdelen Teker, G., Şahin, M. G., & Baytemir, K. (2016). Using generalizability theory to investigate the reliability of peer assessment applications. Journal of Human Sciences, 13(3), 5574-5586. http://dx.doi.org/10.14687/jhs.v13i3.4155
42. Thomson, D., Cummings, E., Ferguson., A. K., Miyuki Moizumi, E., Sher, Y., Wang, X., & Childs, R. A. (2011). A role for research in initial teacher education admissions: A case study from one Canadian university. Canadian Journal of Educational Administration and Policy, 121, 1–23.
43. Tran, T., & Blackman, M. C. (2006). The dynamics and validity of the group selection interview. The Journal of Social Psychology, 146, 183–201. https://10.3200/SOCP.146.2.183-201
44. Webb, N. M., & Shavelson, R. J. (2005). Generalizability theory: overview. Wiley StatsRef: Statistics Reference Online.
45. Wing Tat-Chiu, C. (2001). Scoring performance assessments based on judgements generalizability theory. Springer Science+Business Media, LLC
46. Yılmaz, N.F. & Başbaşa, N.B. (2015). Assessment of sewing and picking skills station reliability with generability theory. Journal of Measurement and Evaluation in Education and Psychology, 6(1): 107-116.
47. Yılmaz, N.F. & Gelbal, S. (2011). Comparison of different patterns with generality theory in the case of communication skills station. Hacettepe University Journal of Education Faculty, 41: 509-518.
48. Yılmaz, N. F. & Tavsancıl, E. (2014). Comparison of balanced and unbalanced patterns in generalizable theory with intramuscular injection station data. Education and Science, 39(175): 285-295.
49. Uzun, N. B., Aktaş, M., Aşiret, S., & Yorulmaz, S. (2018). Using generalizability theory to assess the score reliability of communication skills of dentistry students. Asian Journal of Education and Training, 4(2), 85–90. https://doi.org/10.20448/journal.522.2018.42.85.90