The Evaluation Scale on Vocational English Coursebooks (ESVEC): Construction, validation and implementation

Öz

The main purpose of this study was to develop, validate and implement an instrument to measure the effectiveness of vocational high school English coursebooks. A coursebook is deemed effective if it addresses affirmatively the operational aspects of learning promotion, language capacity building, support allocation, vocational knowledge transfer capacity to target language, and motivation provision to ensure that vocationally aligned foreign language instruction fulfils its goal. Within the framework of the research hereof, sequential exploratory mixed method was adopted. This method encapsulates commencement via a qualitative approach to establish the content and face validity of the scale, pursued by a quantitative approach for the construct validity and reliability of the scale items to be tested. 12 experts comprising 3 vocation-field specialists from the departments of Information Technologies, Health Services and Tourism & Hotel Management and 1 Turkish Language and Literature specialist working in a high school, along with 4 English language teaching specialists, 2 Curriculum and Instruction specialists and 2 Testing and Assessment Specialists participated in the qualitative part of the study while data from 35 students from a local vocational high school in Nevşehir, Turkey, were utilized within the pilot study to test the psychometric properties of ‘Evaluation Scale for Vocational High School English Coursebook’ (ESVEC) on. This pilot application yielded the face and content validity of the scale to be affirmed into its current format. This phase followed the employment of the constructed scale to 420 high school students taking vocational English courses. The scale, processed via the exploratory factor analysis (EFA), yielded a five-factor model. The goodness of fit indices pertinent to this model elicited through confirmatory factor analysis (CFA) upon its application on the same group of students confirmed the fit of the model to the data, with an 18-item scale comprising five factors yielded. This scale was subsequently conducted on 232 IT-department students to evaluate the effectiveness pertinent to the coursebook taught in this vocational English course. The reliability pursuant to ESVEC was justified via multi-analyses.

Anahtar Kelimeler

• Vocational English
• material evaluation
• scale development

Proje Numarası

SBAÜ-2023-12512

Etik Beyan

This research was carried out with the permission of Erciyes University Social and Human Sciences Ethics Committee with the decision number 65 dated 28/02/2023.

Teşekkür

This work has been supported by Erciyes University Scientific Research Projects Coordination Unit with research project SBAÜ-2023-12512.

Mesleki İngilizce Ders Kitaplarını Değerlendirme Ölçeği (ESVEC): Geliştirilmesi, geçerlenmesi ve uygulanması

Öz

Bu çalışmanın temel amacı, meslek liselerinde kullanılan İngilizce ders kitaplarının etkililiğini ölçmeye yönelik bir ölçme aracı geliştirmek, geçerlik ve güvenirlik çalışmalarını yapmak ve bu aracı uygulamaya koymaktır. Bir ders kitabı; öğrenmeyi destekleme, dil becerilerini geliştirme, öğrenme sürecine yönelik destek sunma, mesleki bilgiyi hedef dile aktarabilme ve öğrencinin motivasyonunu artırma gibi boyutları olumlu biçimde ele aldığı ölçüde etkili kabul edilmektedir. Bu unsurların, mesleki odaklı yabancı dil öğretiminin hedeflerine ulaşmasında belirleyici olduğu düşünülmektedir. Araştırmada ardışık keşfedici karma yöntem deseni benimsenmiştir. Bu doğrultuda, ilk aşamada nitel yöntemler kullanılarak ölçeğin kapsam ve görünüş geçerliği sağlanmış, ikinci aşamada ise nicel yöntemlerle yapı geçerliği ve güvenirlik analizleri gerçekleştirilmiştir. Nitel aşamada; Bilişim Teknolojileri, Sağlık Hizmetleri ile Turizm ve Otelcilik alanlarından üç meslek alanı uzmanı, bir lise düzeyinde görev yapan Türk Dili ve Edebiyatı öğretmeni, dört İngilizce öğretimi uzmanı, iki Eğitim Programları ve Öğretim uzmanı ile iki Ölçme ve Değerlendirme uzmanı olmak üzere toplam 12 uzmanın görüşlerine başvurulmuştur. Ölçeğin psikometrik özelliklerini incelemek amacıyla Nevşehir ilinde bulunan bir meslek lisesinde öğrenim gören 35 öğrenciyle pilot uygulama yapılmıştır. Bu pilot çalışma sonucunda, Meslek Liseleri İngilizce Ders Kitaplarını Değerlendirme Ölçeğinin (ESVEC) kapsam ve görünüş geçerliğine sahip olduğu belirlenmiştir. Pilot uygulamanın ardından ölçek, 420 meslek lisesi öğrencisine uygulanmıştır. Yapılan Açımlayıcı Faktör Analizi (AFA) sonucunda ölçeğin beş faktörlü bir yapıya sahip olduğu görülmüş; Doğrulayıcı Faktör Analizi (DFA) ile elde edilen uyum iyiliği indeksleri, bu yapının veriyle uyumlu olduğunu doğrulamıştır. Nihai hâliyle ESVEC, beş faktör altında toplanan 18 maddeden oluşmaktadır. Ölçek, mesleki İngilizce dersi kapsamında kullanılan ders kitabının etkililiğini değerlendirmek amacıyla Bilişim Teknolojileri alanında öğrenim gören 232 öğrenciye uygulanmıştır. Ölçeğin güvenirliği, farklı güvenirlik analizleriyle desteklenmiştir.

Anahtar Kelimeler

• Mesleki İngilizce
• ders materyali değerlendirme
• ölçek geliştirme

Proje Numarası

SBAÜ-2023-12512

Etik Beyan

Bu araştırma Erciyes Üniversitesi Sosyal ve Beşeri Bilimler Etik Kurulunun 28/02/2023 tarih ve 65 sayılı kararıyla izin alınarak yürütülmüştür.

Teşekkür

Bu çalışma Erciyes Üniversitesi Bilimsel Araştırma Projeleri Koordinasyon Birimi tarafından SBAÜ-2023-12512 araştırma projesi ile desteklenmiştir.

Kaynakça

1. Acar-Güvendir, M., & Özer-Özkan, Y. (2022). Item removal strategies conducted in exploratory factor analysis: A comparative study. International Journal of Assessment Tools in Education, 9 (1), 165-180. https://doi.org/10.21449/ijate.827950
2. Adamson, H. D. (2022). Specialized languages activate distinct semantic and lexical features compared to general language: An empirical investigation. Languages, 7(3), Article 89. https://doi.org/10.3390/languages7030089
3. Agzamovna, T. U. (2024). Bridging language and professional skills: effective ESP pedagogy for diverse learner needs. Current Research Journal of Pedagogics, 5(10), 36-41. https://doi.org/10.37547/pedagogics-crjp-05-10-06
4. Al-Lawati, N. (2016). Multi-disciplinary English for specific purposes (ESP) courses: A new dimension to creative English language teaching (ELT). English for Specific Purposes World, 17 (49), 1-16.
5. Antoniou, F., & Alghamdi, M. H. (2024). Confidence in mathematics is confounded by responses to reverse coded items. Frontiers in Psychology. Advance online publication. https://doi.org/10.3389/fpsyg.2024.1489054
6. Avcı, N., & Engin-Demir, C. (2021). Promoting the vocational English language competences of students in an English for specific purposes context: A needs assessment. Psycho-Educational Research Reviews, 10 (3), 125-139. https://doi.org/10.52963/PERR_Biruni_V10.N3.08
7. Basturkmen, H. (2010). Developing courses in English for specific purposes. Palgrave Macmillan.
8. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149

9. Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York: The Guilford Press.
10. Bulut, H. Ç., Bulut, O., & Clelland, A. (2024). A psychometric network‐analysis approach for detecting item wording effects in self‐report measures. Field Methods, 37(1), 20–36. https://doi.org/10.1177/1525822X241247444
11. Can, A. (2016). SPSS ile bilimsel araştırma sürecinde nicel veri analizi [Quantitative data analysis in the process of scientific research with SPSS]. Ankara: Pegem Akademi.
12. Cai, X., & Yuan, K. (2021). A method for splitting a sample into equivalent subsamples in factor analysis. Behavior Research Methods, 54(6), 2881–2897. https://doi.org/10.3758/s13428-021-01750-y
13. Clark, L.A. & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7 (3), 309-319. https://doi.org/10.1037/1040-3590.7.3.309
14. Coxhead, A., Parkinson, J., Mackay, J., McLaughlin, E. (2019). English for vocational purposes: Language use in trades education. London: Routledge, Taylor & Francis Group.
15. Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Los Angeles: SAGE Publications.
16. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
17. Cunningsworth, A. (1995). Choosing your coursebook. Oxford: Heineman.
18. Damayanti, D., Fauzi, A. & Inayati, A.M. (2018). Learning materials: The “nucleus” of language teaching. Journal of English Education, 3, 1-8. https://doi.org/10.31327/jee.v3i1.417
19. Dassanayake, N. (2022). Development of an evaluation checklist for localized Chinese language textbooks in Sri Lanka. Lingua Cultura, 16(2), 257–269. https://doi.org/10.21512/lc.v16i2.8778
20. DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Thousand Oaks, CA: Sage Publications.
21. Dudley-Evans, T. & St. John, M. J. (2001). Developments in English for specific purposes: A multidisciplinary approach. Cambridge: Cambridge University Press.
22. Dülger, O. (2016). Evaluation of EFL coursebooks taught in Turkey based on teachers’ views. Journal of Advances in English Language Teaching, 4(1), 1-11.
23. Ellederová, E. (2019). Design-based research of an ESP coursebook: Evaluation of the coursebook pilot version. Folio–Journal of the Materials Development Association, 19(1), 32-39.
24. Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). Newbury Park: SAGE Publications.
25. Garton, S. & Graves, K. (2014). International perspectives on materials in ELT. Basingstoke: Palgrave Macmillan.
26. Ghafar, Z. N. (2022). English for specific purposes in English language teaching: Design, development, and environment-related challenges: An overview. Canadian Journal of Language and Literature Studies, 2(6), 32–42. https://doi.org/10.53103/cjlls.v2i6.72
27. Golonka, E. M., Bowles, A. R., Frank, V. M., Richardson, D. L., & Freynik, S. (2014). Technologies for foreign language learning: A review of technology types and their effectiveness. Computer Assisted Language Learning, 27*(1), 1-36. https://doi.org/10.1080/09588221.2012.700315
28. Graziotin, D., Lenberg, P., & Feldt, R. (2020). Psychometrics in behavioral software engineering: A methodological introduction with guidelines. arXiv. https://doi.org/10.48550/arXiv.2005.09959
29. Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2019). Multivariate data analysis (8th ed.). England: Pearson Prentice.
30. Hinkin, T.R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1 (1), 104-121. https://doi.org/10.1177/109442819800100106
31. Huang, S. H., & Yang, L. C. (2020). English for general purposes (egp) at a taiwanese university: A mix-methodological analysis of students' perspectives. 清華教育學報, 37(1), 41-83. https://doi.org/10.6869/THJER.202006_37(1).0002
32. İbrahim, H.H., (2019). The role of an ideal ESP practitioner. Open Journal of Modern Linguistics, 9(2), 76-91. https://doi.org/10.4236/ojml.2019.92008
33. İlhan, M., Güler, N., Taşdelen Teker, G., & Ergenekon, Ö. (2024). The effects of reverse items on psychometric properties and respondents’ scale scores according to different item reversal strategies. International Journal of Assessment Tools in Education, 11(1), 20–38. https://doi.org/10.21449/ijate.1345549
34. Islam, M. (2015). The differences and similarities between English for specific purposes (ESP) and English for general purposes (EGP) teachers. Journal of Research in Humanities, 51(1), 1-11.
35. Kaiser, H.F. (1974) An index of factorial simplicity. Psychometrika, 39, 31-36. http://dx.doi.org/10.1007/BF02291575
36. Kalfut, T. (2024). Approaches and limitations of L2 textbook evaluation research. International Journal of Language and Literary Studies, 6(1), 216–225. https://doi.org/10.36892/ijlls.v6i1.1617
37. Kamolovna, N. Z. (2022). Differences and similarities between English for specific purposes (ESP) and English for general purposes (EGP) teachers. British View, 7(4), 56-62. https://doi.org/10.5281/zenodo.7313244
38. Karagöz, Y., & Bardakçı, S. (2020). Bilimsel araştırmalarda kullanılan ölçme araçları ve ölçek geliştirme. Ankara: Nobel akademik yayıncılık.
39. Kaya, S. (2021). From Needs Analysis to Development of a Vocational English Language Curriculum: A Practical Guide for Practitioners. Journal of Pedagogical Research, 5(1), 154-171. https://doi.org.10.33902/JPR.2021167471
40. Kline, R. B. (2023). Principles and practice of structural equation modeling (5th ed.). Guilford Press.
41. Koo, M., & Yang, S.-W. (2025). Likert-type scale. Encyclopedia, 5, 18. https://doi.org/10.3390/encyclopedia5010018
42. Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Pyschology, 28 (4), 563-575.
43. Liu, D. Y., & Zhang, P. (2024). A needs analysis of ESP courses in colleges of art and design: Consensus and divergence. PLOS ONE, 19(6), e0305210. https://doi.org/10.1371/journal.pone.0305210
44. McDonough, J. & Shaw, C. (2003). Materials and methods in ELT. Blackwell.
45. Metin, M. (2014). Kuramdan uygulamaya eğitimde bilimsel araştırma yöntemleri. Pegem Yayınları.
46. Mithans, J., & Grmek, M. (2020). The use of textbooks in the teaching–learning process: A curriculum vehicle for novice teachers. Journal of Curriculum Studies, 12(1), 45–63. https://doi.org/10.12345/jcs.2020.0012
47. Nimechisalem, V. & Mukundan, J., (2015). Refinement of the English language teaching textbook evaluation checklist. Pertanika Journal of Social Science and Humanities,23(4), 761-780.
48. Öz, Ö. (2019). A post-use evaluation of an EFL coursebook from the perspectives of preparatory school students and instructors: A mixed methods study (Master’s thesis, Middle East Technical University). METU Open Access Theses and Dissertations.  https://files.eric.ed.gov/fulltext/EJ1273168.pdf
49. Özgür, A., Altun, A., & Akar, S. G. M. (2020). Çokluortam öğrenme materyaline bağlanma ölçeği: Geliştirme, geçerlik ve güvenirlik çalışmasi. Eğitim Teknolojisi Kuram ve Uygulama, 10(2), 321-344. https://doi.org/10.17943/etku.625811
50. Pan, F. (2024). Enhancing learners’ language autonomy: Student-centered approaches in the classroom. Transactions on Social Science Education and Humanities Research, 4, 74–80. https://doi.org/10.62051/v3988j29
51. Robinson, P. (1991). ESP today: A practitioner’s guide. New York: Prentice Hall.
52. Romero Jeldres, M., Diaz Costa, E., & Faouzi Nadim, T. (2023). A review of Lawshe’s method for calculating content validity in the social sciences. Frontiers in Education, 8, Article 1271335. https://doi.org/10.3389/feduc.2023.1271335
53. Schneeberger, A. (2021). Language proficiency as a gatekeeper to vocational education: The case of young migrants. Journal of Vocational Education & Training, 73(4), 649–667. https://doi.org/10.1080/13636820.2021.1872531
54. Schumacker, R. E., & Lomax, R. G. (2016). A beginner’s guide to structural equation modeling (4th ed.). New York: Routledge.
55. Sekhar, G. & Chakravorty, Sujata. (2017). English for special or specific purposes (ESP): A study on teaching ESP courses. ACADEMICIA: An International Multidisciplinary Research Journal. 7(4), 112. https://doi.org/10.5958/2249-7137.2017.00041.6.
56. Shak, M. S. Y., Albakri, I. S. M. A., Shukor, S. S., & Tahir, M. H. M. (2022). Criteria in English language textbook evaluation checklists: A systematic review. IARTEM E Journal, 14(1), 1–19. https://doi.org/10.21344/iartem.v14i1.978
57. Shkeer, A. S., & Awang, Z. (2019). Exploring the items for measuring the marketing information system construct: An exploratory factor analysis. International Review of Management and Marketing, 9(6), 87–97. https://doi.org/10.32479/irmm.8622
58. Solis Salazar, M. (2015). The dilemma of combining positive and negative items in scales. Psicothema, 27(2), 192-200. https://doi.org/10.7334/psicothema2014.266
59. Strack, R., Booker, M., von der Linden, C., & Strohmayr, A. (2014). Decoding global talent. The Boston Consulting Group and Network.
60. Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99–103.
61. Synowiec, C., Fletcher, E., Heinkel, L., & Salisbury, T. (2023). Getting rigor right: a framework for methodological choice in adaptive monitoring and evaluation. Global Health: Science and Practice, 11(Supplement 2). https://doi.org/10.9745/GHSP-D-22-00243.
62. Tabachnick, B. G., & Fidell, L. S. (2021). Using multivariate statistics (7th ed., updated ed.). Pearson.
63. Tomlinson, B. & Masuhara, H. (2018). The complete guide to the theory and practice of materials development for language learning. Wiley Blackwell
64. Yurdabakan, İ. & Çüm, S. (2017). Scale development in behavioral sciences (Based on exploratory factor analysis). Turkish Journal of Family Medicine and Primary Care, 11(2), 108-126. https://doi.org/10.21763/tjfmpc.317880
65. Yıldırım, N. K., & Bal, N. G. (2023). English language education in a vocational school: A qualitative case study. Participatory Educational Research, 10(2), 275-297. https://doi.org/10.17275/per.23.40.10.2
66. Yıldırım, M., & Ogur, E. (2023). Dijital hikâye kullanımına yönelik tutum ölçeği (DHKYTÖ): Ölçek geliştirme çalışması. Korkut Ata Türkiyat Araştırmaları Dergisi, (12), 1113-1135. https://doi.org/10.51531/korkutataturkiyat.1351729
67. Zhang, X., & Savalei, V. (2016). Improving the factor structure of psychological scales: The expanded format as an alternative to the Likert scale format. Behavior Research Methods, 48(3), 1191–1205. https://doi.org/10.3758/s13428-015-0624-4
68. Zhou, X., & Liu, Y. (2022). Evaluating the split data strategy for factor analysis: A Monte Carlo simulation study. Behavior Research Methods, 54(5), 2104–2122. https://doi.org/10.3758/s13428-021-01735-4
69. Zokaeieh, S., Najafi Karimi, S., Nouri, F., & Hakimzadeh, A. (2019). Developing a critical checklist for textbook evaluation. The Journal of English Language Pedagogy and Practice, 12(24), 202–225. https://doi.org/10.30495/JAL.2019.671933