Reliability and Validity of Instruments Measuring English Teachers’ TPACK

Abstract

In this study, issues of validity and reliability of a wide range of instruments used to measure technological pedagogical content knowledge level of English teachers were discussed. To this end, the search in the databases of ERIC, ScienceDirect, Scopus, EBSCOhost, and Web of Science was conducted. As a result of applying a set of criteria to publications retrieved from the databases, 60 studies (including 40 articles, 14 dissertations and 6 conference papers) were found suitable for analysis in the current study. A two-level analysis was conducted. First one was study-level analysis focusing on general characteristics of each study and the second one was an instrument-level analysis that focuses on target audience and research instruments. As a consequence of the analysis at the study-level, 128 instruments were classified into five types of instruments including, open-ended questionnaire, observation, performance assessment, interview, and self-report instruments. At the instrument-level analysis, issues of validity and reliability of those instruments and target audience were investigated. The findings revealed that 60% of the reviewed studies did not provide any index of reliability, and similarly over 80% of the studies presented no evidence of validity.

Keywords

• reliability
• validity
• TPACK
• English teachers

References

1. Abera, B. (2014). Applying a technological pedagogical content knowledge framework in Ethiopian English language teacher education. In Multicultural Awareness and Technology in Higher Education: Global Perspectives, 286-301. IGI Global.
2. Agyei, D. D., &Voogt, J. (2012). Developing technological pedagogical content knowledge in pre-service mathematics teachers through collaborative design. Australasian journal of educational technology, 28(4), 547-564.
3. Alahmari, A. S. (2013). An investigation of Saudi Arabian EFL teachers' engagement with technology. (Unpublished doctoral dissertation). Retrieved from https://monash.figshare.com/ 4701100_monash_120645.pdf
4. Alhababi, H. H. (2017). Technological pedagogical content knowledge (TPACK) effectiveness on English teachers and students in Saudi Arabia. (Unpublished doctoral dissertation). Retrieved from https://digscholarship.unco.edu/dissertations/456/
5. Ansyari, M. H. (2012). The development and evaluation of a professional development arrangement for technology integration to enhance communicative approach in English language teaching (unpublished master's thesis). Retrieved from https://essay.utwente.nl/62294/1/MSc_Ansyari_M.F._-_S1084712.pdf
6. Ansyari, M. F. (2015). Designing and evaluating a professional development programme for basic technology integration in English as a foreign language (EFL) classrooms. Australasian Journal of Educational Technology, 31(6), 699-712.
7. Archambault, L., & Crippen, K. (2009). Examining TPACK among K-12 online distance educators in the United States. Contemporary issues in technology and teacher education, 9(1), 71-88.
8. Aşık, A., Eroğlu İnce, B., & Şarlanoğlu Vural, A. (2018). Investigating learning technology by design approach in pre-service language teacher education: Collaborative and reflective experiences. Eğitimde Nitel Araştırmalar Dergisi,6(1), 37-53.

9. Baser, D., Kopcha, T. J., &Ozden, M. Y. (2016). Developing a technological pedagogical content knowledge (TPACK) assessment for preservice teachers learning to teach English as a foreign language. Computer Assisted Language Learning, 29, 749–764.
10. Berg, B. L. (2001). Qualitative research methods for the social sciences (4th ed.). Boston, MA: Allyn and Bacon.
11. Bordens, K. S., & Abbott, B. B. (2002). Research design and methods: A process approach. McGraw-Hill.
12. Bowers, J. S., & Stephens, B. (2011). Using technology to explore mathematical relationships: A framework for orienting mathematics courses for prospective teachers. Journal of Mathematics Teacher Education, 14(4), 285-304.
13. Canbazoglu-Bilici, S., Yamak, H., Kavak, N., & Guzey, S.S. (2013). Technological pedagogical content knowledge self-efficacy scale (TPACK-SeS) for pre-service science teachers: construction, validation, and reliability. Eurasian Journal of Educational Research, 52, 37– 60.
14. Chai, C. S., Koh, J. H. L., & Tsai, C.-C. (2010). Facilitating preservice teachers' development of technological, pedagogical, and content knowledge (TPACK). Educational Technology & Society, 13(4), 63–73.
15. Chai, C. S., Chin, C. K., Koh, J. H. L., & Tan, C. L. (2013). Exploring Singaporean Chinese language teachers’ technological pedagogical content knowledge and its relationship to the teachers’ pedagogical beliefs. The Asia-Pacific Education Researcher, 22(4), 657-666.
16. Chai, C. S., Koh, J. H. L., & Tsai, C. C. (2013). A review of technological pedagogical content knowledge. Journal of Educational Technology & Society, 16(2), 31-51.
17. Chewning, R. (2015). Secondary English teachers’ dispositions toward technology integration in one-to-one environments (Unpublished Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Global database. (UMI No. 3745350)
18. Cohen, L. (2007). Experiments, quasi-experiments, single-case research and meta-analysis (Cohen, L., Manion, L., & Morrison, K. in Eds) Research methods in education. (6th ed.).
19. Cox, S., & Graham, C. R. (2009). Using an elaborated model of the TPACK framework to analyse and depict teacher knowledge. TechTrends, 53(5), 60-69.
20. Creswell, J. W., &Poth, C. N. (2016). Qualitative inquiry and research design: Choosing among five approaches. Sage publications.
21. Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Pearson Education, Inc.
22. Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Sage Publications.
23. DeMonbrun, M. R. M., Finelli, C. J., & Shekhar, P. (2015). Methods for establishing validity and reliability of observation protocols. In ASEE Annual Conference and Exposition, Conference Proceedings (Vol. 122, No. 122, pp. 1-10).
24. Dörnyei, Z. (2007). Research Methods in Applied Linguistics: Quantitative Qualitative, and Mixed Methodologies. Oxford: Oxford University Press.
25. Drajati, N. A., Tan, L., Haryati, S., Rochsantiningsih, D., &Zainnuri, H. (2018). Investigating English language teachers in developing TPACK and multimodal literacy. Indonesian Journal of Applied Linguistics, 7(3), 575-582.
26. Dunlap, J. C. (2006). Using guided reflective journaling activities to capture students’ changing perceptions. Techtrends: Linking Research & Practice to Improve Learning, 50, 20–26. https://doi.org/10.1007/s11528-006-7614-x
27. Ersanli, C. Y. (2016). Improving technological pedagogical content knowledge (TPACK) of pre-service English language teachers. International Education Studies, 9(5), 18-27.
28. Frey, B. B. (2013). Modern classroom assessment. Sage publications.
29. Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational Research: An Introduction. New York: Person Education.
30. Gardner, F. (2000). Methodological issues in the direct observation of parent–child interaction: Do observational findings reflect the natural behavior of participants?. Clinical child and family psychology review, 3(3), 185-198.
31. Gibbs L., Kealy M., Willis K., Green J., Welch N. & Daly J. (2007) What have sampling and data collection got to do with good qualitative research? Australian and New Zealand Journal of Public Health, 31(6), 540 544. https://doi.org/10.1111/j.1753 6405.2007.00140.x
32. Graham, C. R. (2011). Theoretical considerations for understanding technological pedagogical content knowledge (TPACK). Computers & Education, 57(3), 1953-1960.
33. Graham, C. R., Borup, J., & Smith, N. B. (2012). Using TPACK as a framework to understand teacher candidates' technology integration decisions. Journal of Computer Assisted Learning, 28(6), 530-546.
34. Good, T. L. (Ed.). (2008). 21st-century education: A reference handbook. Vol.1. Sage.
35. Harriman, C. L. S. (2011). The impact of TPACK and digital storytelling as a learning experience for pre-service teachers in a learning-by-designing project (unpublished doctoral dissertation). University of Georgia, USA.
36. Harris, J., Grandgenett, N., & Hofer, M. (2010, March). Testing a TPACK-based technology integration assessment rubric. In Society for Information Technology & Teacher Education International Conference, 3833-3840. Association for the Advancement of Computing in Education (AACE).
37. Jang, S. J., & Tsai, M. F. (2012). Exploring the TPACK of Taiwanese elementary mathematics and science teachers with respect to use of interactive whiteboards. Computers & Education, 59(2), 327–338.
38. Jen, T. H., Yeh, Y. F., Hsu, Y. S., Wu, H. K., & Chen, K. M. (2016). Science teachers’ TPACK-Practical: Standard-setting using an evidence-based approach. Computers & Education, 95, 45–62.
39. Johnson, R. B., & Christensen, L. (2019). Educational research: Quantitative, qualitative, and mixed approaches. SAGE publications.
40. Johnson, R. B., Christensen, L. B., & Turner, L. A. (2014). Research methods, design and analysis. Pearson Education.
41. Joppe, M. (2000). The Research Process: Tests and Questionnaires. Quantitative Applications in the Social Sciences, 211-236.
42. Kafyulilo, A., Fisser, P., Pieters, J., &Voogt, J. (2015). ICT use in science and mathematics teacher education in Tanzania: Developing technological pedagogical content knowledge. Australasian Journal of Educational Technology, 31(4), 381–394.
43. Kitto S. C., Chesters J. & Grbich C. (2008) Quality in qualitative research. Medical Journal of Australia, 188(4), 243-246.
44. Kramarski, B., & Michalsky, T. (2010). Preparing preservice teachers for self-regulated learning in the context of technological pedagogical content knowledge. Learning and Instruction, 20(5), 434-447.
45. Kharade, K., &Peese, H. (2014). Problem-based learning: A promising pathway for empowering pre-service teachers for ICT-mediated language teaching. Policy Futures in Education, 12(2), 262-272.
46. Koçoğlu, Z. (2009). Exploring the technological pedagogical content knowledge of pre-service teachers in language education. Procedia-Social and Behavioural Sciences, 1(1), 2734-2737.
47. Koehler, M. J., & Mishra, P. (2008). Introducing TPCK. AACTE Committee on Innovation and Technology (Ed.), The handbook of technological pedagogical content knowledge (TPCK) for educators, 3-29. Lawrence Erlbaum Associates.
48. Koehler, M., & Mishra, P. (2009). What is technological pedagogical content knowledge (TPACK)? Contemporary issues in technology and teacher education, 9(1), 60-70.
49. Koehler, M. J., Shin, T. S., & Mishra, P. (2012). How do we measure TPACK? Let me count the ways. In educational technology, teacher knowledge, and classroom impact: A research handbook on frameworks and approaches, 16-31. IGI Global.
50. Koh, J. H. L., Chai, C. S., & Tsai, C. C. (2013). Examining practising teachers’ perceptions of technological pedagogical content knowledge (TPACK) pathways: A structural equation modelling approach. Instructional Science, 41(4), 793-809.
51. Kurt, G., Akyel, A., Koçoğlu, Z., & Mishra, P. (2014). TPACK in practice: A qualitative study on technology integrated lesson planning and implementation of Turkish pre-service teachers of English. ELT Research Journal, 3(3), 153-166.
52. Kwangsawad, T. (2016). Examining EFL Pre-service Teachers' TPACK Trough Self-report, Lesson Plans and Actual Practice. Journal of Education and Learning, 10(2), 103-108.
53. Landry, G. A. (2010). Creating and validating an instrument to measure middle school mathematics teachers’ technological pedagogical content knowledge. (TPACK). (Unpublished doctoral dissertation). University of Tennessee. Retrieved from http://trace.tennessee.edu/utk_graddiss/720.
54. Lavrakas, P. J. (2008). Encyclopaedia of survey research methods. Sage publications.
55. Le, N., & Song, J. (2018). TPACK in a CALL course and its effect on Vietnamese pre-service EFL teachers. The Asian EFL Journal Quarterly, 31.
56. Lewis-Beck, M., Bryman, A. E., & Liao, T. F. (2004). The Sage encyclopaedia of social science research methods. Sage Publications.
57. Liu, M. H., & Kleinsasser, R. (2015). Exploring EFL teachers’ knowledge and competencies: In-service program perspectives. Language Learning & Technology, 19(1), 119-138.
58. Luppicini, R. (2005). A systems definition of educational technology in society. Educational Technology & Society, 8(3), 103-109.
59. Maxwell, J. A. (2012). Qualitative research design: An interactive approach. Sage.
60. Merriam, S. B., & Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation. John Wiley & Sons.
61. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers’ college record, 108(6), 1017-1054.
62. Mouza, C., Karchmer-Klein, R., Nandakumar, R., Ozden, S. Y., & Hu, L. K. (2014). Investigating the impact of an integrated approach to the development of preservice teachers’ technological pedagogical content knowledge (TPACK). Computers & Education, 71, 206–221.
63. Niess, M. L. (2011). Investigating TPACK: Knowledge growth in teaching with technology. Journal of educational computing research, 44(3), 299-317.
64. Nunan, D. (1999). Research methods in language learning (8th printing). Cambridge: CUP.
65. O’Connell, T. S., &Dyment, J. E. (2011). The case of reflective journals: Is the jury still out? Reflective Practice, 12, 47–59. https://doi.org/10.1080/14623943.2011.541093
66. Oz, H. (2015). Assessing pre-service English as a foreign language teachers' technological pedagogical content knowledge. International Education Studies, 8(5), 119-130.
67. Patton, M. Q. (2001). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.
68. Richards, J. C. (2002). Longman Language Teaching and Applied Linguistics. Pearson Education.
69. Rubadeau, Z. (2016). An exploration of English language teacher educators’ cognitions and practices in relation to the pedagogical purposes and efficacies of 21st-century digital technologies. Durham theses, Durham University. Available at Durham E-Theses Online: http://etheses.dur.ac.uk/11506/
70. Saldana, J. (2011). Fundamentals of qualitative research. OUP USA.
71. Seawright, J. (2016). Multi-methodsocialscience: Combiningqualitativeandquantitativetools. Cambridge University Press.
72. Sessoms, D. (2008). Interactive instruction: Creating interactive learning environments through tomorrow’s teachers. International Journal of Technology in Teaching and Learning, 4(2), 86-96.
73. Schmidt, D.A., Baran, E., Thompson, A.D., Mishra, P., Koehler, M.J., & Shin, T.S. (2009). Technological pedagogical content knowledge (TPACK): The development and validation of an assessment instrument for pre-service teachers. Journal of Research Technology in Education, 42(2), 123–149.
74. Shin, T., Koehler, M., Mishra, P., Schmidt, D., Baran, E., & Thompson, A. (2009, March). Changing technological pedagogical content knowledge (TPACK) through course experiences. In Society for Information Technology & Teacher Education International Conference, 4152-4159. Association for the Advancement of Computing in Education (AACE).
75. Tai, S. J. D. (2015). From TPACK-in-action workshops to classrooms: CALL competency developed and integrated. Language Learning & Technology, 19(1), 139-164.
76. Teijlingen, van, E., & Hundley, V. (2001). The importance of pilot studies. Social Research Update, 35. Retrieved May 2, 2020, from http://www.soc.surrey.ac.uk/sru/SRU35.html.
77. Tokmak, H. S., Incikabi, L., & Ozgelen, S. (2013). An investigation of change in mathematics, science, and literacy education pre-service teachers’ TPACK. The Asia-Pacific Education Researcher, 22(4), 407-415.
78. Tschannen-Moran, M., & Hoy, A. W. (2001). Teacher efficacy: Capturing an elusive construct. Teaching and teacher education, 17(7), 783-805.
79. Tseng, J. J., Cheng, Y. S., & Yeh, H. N. (2019). How pre-service English teachers enact TPACK in the context of web-conferencing teaching: A design thinking approach. Computers & Education, 128, 171-182.
80. Turgut, Y. (2017a). Tracing pre-service English language teachers’ perceived TPACK in sophomore, junior, and senior levels. Cogent Education, 4(1368612), 1–20.
81. Turgut, Y. (2017b). A comparison of pre-service, in-service and formation program for teachers' perceptions of technological pedagogical content knowledge (TPACK) in English language teaching (ELT). Educational Research and Reviews, 12(22), 1091-1106.
82. Vereshchahina, T., Liashchenko, O., & Babiy, S. (2018). English language teachers’ perceptions of hybrid learning at university level. Advanced Education, 5(10), 88-97.
83. Young, J. R., Young, J. L., & Hamilton, C. (2013). The use of confidence intervals as a meta-analytic lens to summarise the effects of teacher education technology courses on preservice teacher TPACK. Journal of Research on Technology in Education, 46(2), 149-172.
84. Wang, A. Y. (2016, June). TPACK assessment in English language arts for teachers of English as a foreign language. In EdMedia+ Innovate Learning,1082-1087. Association for the Advancement of Computing in Education (AACE).
85. Weerakanto, P. (2019). Digital literacies of English language teachers and students and their perceptions of technology-enhanced language learning and teaching in Thailand (unpublished doctoral dissertation). The University of Arizona. Retrieved from https://repository.arizona.edu/handle/10150/633068.
86. Wetzel, K., & Marshall, S. (2011). TPACK goes to sixth grade: Lessons from a middle school teacher in a high-technology-access classroom. Journal of Digital Learning in Teacher Education, 28(2), 73-81.
87. Winter, G. (2000). A comparative discussion of the notion of validity in qualitative and quantitative research. The qualitative report, 4(3), 1-14.
88. Wu, Y. T., & Wang, A. Y. (2015). Technological, pedagogical, and content knowledge in teaching English as a foreign language: Representation of primary teachers of English in Taiwan. The Asia-Pacific Education Researcher, 24(3), 525-533.