A rating scale development study for the evaluation of lesson plans and teaching practices on argumentation-based inquiry

Abstract

The purpose of this study was to develop a reliable and valid rating scale for the use of the assessment and evaluation of lesson plans and teaching practices that are based on argumentation-based inquiry (ABI). The study covered two academic years (four academic semesters). Qualitative and quantitative methods were utilized throughout the development of the rating scale including data collection and data analyses. A purposive sample of 72 pre-service science teachers (PSTs) who were enrolled in a public university located in East Black Sea region of Turkey constituted the sample of the study. Content Validity Ratio (CVR=.80) and Content Validity Index (CVI=.94) values were calculated as measures of content validity. Pearson Correlation Coefficient (r=.96) and Cohen’s Kappa value (κ value was between .60 and 1.00) were calculated to test inter-rater reliability of the scores obtained by the rating scale. Findings provided evidence for the reliability and the validity of the ABI rating scale. ABI lesson plan template and ABI rating scale developed for the assessment and evaluation of ABI lesson plans and subsequent teaching practices are provided to the readers. Contributions to the field are discussed.

Keywords

• Argumentation based inquiry
• Rating scale
• Science writing heuristic
• Lesson plans

References

1. Akkus, R., Günel, M., & Hand, B. (2007). Comparing an inquiry-based approach known as the science writing heuristic to traditional science teaching practices: Are there differences? International Journal of Science Education, 14(1), 1745 1765. https://doi.org/10.1080/09500690601075629
2. Aktamış, H., & Atmaca, A.C. (2016). Fen bilgisi öğretmen adaylarının argümantasyon tabanlı öğrenme yaklaşımına yönelik görüşleri [View’s of pre service science teachers about argumentation based learning approach]. Electronic Journal of Social Sciences, 15(58), 936–947. http://doi.org/10.17755/esosder.48760
3. Arık, M., & Akçay, B. (2017). Argümantasyon tabanlı öğrenme [Argumantation based learning]. In B. Akcay (Ed.), Fen bilimleri eğitimi alanındaki öğrenme ve öğretme yaklaşımları [Learning and teaching approaches in science education] (pp.177 192). Pegem A Yayıncılık.
4. Australian Curriculum, Assessment and Reporting Authority. (2012). The Australian curriculum: Science (Version 3.0). Commonwealth of Australia, NSW.
5. Aydeniz, M., & Ozdilek, Z. (2016). Assessing and enhancing pre service science teachers’ self efficacy to teach science through argumentation: Challenges and possible solutions. International Journal of Science and Mathematics Education, 14(7), 1255 1273. http://doi.org/10.1007/s10763-015-9649-y
6. Ayre, C., & Scally, A.J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79 86. https://doi.org/10.1177/0748175613513808
7. Bean, J.C. (1996). Engaging ideas: The professor’s guide to integrating writing, critical thinking, and active learning in the classroom. Jossey-Bass Press.
8. Bell, P., & Linn, M.C. (2000). Scientific arguments as learning artifacts: Designing for learning from the web with KIE. International Journal of Science Education, 22(8), 797 817. https://doi.org/10.1080/095006900412284

9. Berland, L.K., & McNeill, K.L. (2010). A learning progression for scientific argumentation: Understanding student work and designing supportive instructional contexts. Science Education, 94(5), 765 793. https://doi.org/10.1002/sce.20402
10. Burke, K.A., Greenbowe, T.J., & Hand, B.M. (2006). Implementing the science writing heuristic in the chemistry laboratory. Journal of Chemical Education, 83(7), 1032 1038. https://doi.org/10.1021/ed083p1032
11. Buyukozturk, Ş., Kılıc-Cakmak, E., Akgun, Ö.E., Karadeniz, S. & Demirel, F. (2020). Bilimsel araştırma yöntemleri [Scientific research methods]. (29th ed.). Pegem Yayıncılık.
12. Cavagnetto, A.R., Hand, B., & Norten Meier, L. (2010). Negotiating the inquiry question: A comparison of whole class and small group strategies in grade five science classrooms. Research in Science Education, 41(2), 193 209. https://doi.org/10.1007/s11165 009 9152-y
13. Chan, J., Fancourt, N., & Guilfoyle, L. (2021). Argumentation in religious education in England: An analysis of locally agreed syllabuses. British Journal of Religious Education, 43(4), 458 471. https://doi.org/10.1080/01416200.2020.1734916
14. Chan, J., & Erduran, S. (2022). The impact of collaboration between science and religious education teachers on their understanding and views of argumentation. Research in Science Education, 1-17. https://doi.org/10.1007/s11165-022-10041-1
15. Chen, C.H., & She, C. (2012). The impact of recurrent online synchronous scientific argumentation on students’ argumentation and conceptual change. Educational Technology & Society, 15(1), 197-210. https://www.jstor.org/stable/jeductechsoci.15.1.197
16. Chin, C., & Osborne, J. (2010). Students’ questions and discursive ınteraction: their ımpact on argumentation during collaborative group discussions in science. Journal of Research in Science Teaching, 47(7), 883 908. https://doi.org/10.1002/tea.20385
17. Choi, A., Notebaert, A., Diaz, J., & Hand, B. (2010). Examining arguments generated by year 5, 7, and 10 students in science classrooms. Research in Science Education, 40(2), 149 169. https://doi.org/10.1007/s11165-008-9105-x
18. Cirit Gül, A., Apaydın, Z., & Çobanoğlu, E.O., (2021). Türkiye'de Argümantasyon ile ilgili yapılan lisansüstü tezlerin incelenmesi [Investigation of graduate thesis about argumentation in Turkey: Thematic Content Analysis]. Ondokuz Mayıs University Journal of Faculty of Education, 40(2), 591-628. https://doi.org/10.7822/omuefd.863712
19. Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). SAGE Publications
20. Cresswell, J.W., & Plano Clark, V.L. (2011). Designing and conducting mixed method research (2nd ed.). Thousand Oaks.
21. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
22. Çepni, S., & Çil, E. (2016). Fen bilimleri dersi öğretim programı (Tanıma, Planlama, Uygulama ve TEOG ile ilişkilendirme). İlkokul ve ortaokul öğretmen el kitabı [Science lesson instruction program (Familiarity, Planning, Application and Relating with TEOG). Primary and middle school teacher handbook]. Pegem.
23. Çoban, G.Ü., Akpınar, E., Baran, B., Sağlam, M.K., Özcan, E., & Kahyaoğlu, Y. (2016). The evaluation of "technological pedagogical content knowledge based argumentation practices" training for science teachers. Education & Science/Egitim ve Bilim, 41(188), 1 33. https://doi.org/10.15390/EB.2016.6615
24. Driver, R., Newton, P., & Osborne, J. (2000). Establishing the norms of scientific argumentation in classrooms. Science Education, 84(3), 287 312. https://doi.org/10.1002/(SICI)1098-237X(200005)84:3<287::AID-SCE1>3.0.CO;2-A
25. Ecevit, T., & Kaptan, F. (2021). Describing the argument based inquiry teaching model designed for gaining the 21st century skills. Hacettepe University Journal of Education, 36(2), 470-488. https://doi.org/10.16986/HUJE.2019056328
26. Erduran, S., Ardaç, D., & Yakmacı Guzel, B. (2006). Promoting argumentation in preservice teacher education in science. Eurasia Journal of Mathematics, Science and Technology Education, 2(2), 1 14.
27. Erduran, S., & Jimenez Aleixandre, P. (2007). Argumentation in science education: Perspectives from classroom-based research. Springer Science.
28. Erduran, S., Simon, S., & Osborne, J. (2004). Tapping into argumentation: Developments in the application of Toulmin’s argument pattern for studying science discourse. Science Education, 88(6), 915 933. https://doi.org/10.1002/sce.20012
29. Fraenkel, J.R., Wallen, N., & Hyun, H. (2012). How to design and evaluate research in education (8th ed.). McGraw Hill.
30. Frazel, M. (2010). Digital storytelling guide for educators. International Society for Technology in Education (ISTE).
31. Gencel, İ.E., & Ilıman, M. (2019). Argümantasyona dayalı öğretime ilişkin bir durum çalışması [A case study on argumentation based teaching]. International Journal of Curriculum and Instructional Studies, 9(1), 53-72. https://doi.org/10.31704/ijocis.2019.003
32. Guilfoyle, L., Erduran, S. & Park, W. (2021) An investigation into secondary teachers’ views of argumentation in science and religious education, Journal of Beliefs & Values, 42(2), 190-204. https://doi.org/10.1080/13617672.2020.1805925
33. Gunel, M., Kingir, S., & Geban, Ö. (2012). Analyses of argumentation and questioning patterns in argument-based inquiry classrooms. Egitim ve Bilim, 37(164), 316-330.
34. Hand, B. (2008). Introducing the science writing heuristic approach. In B. Hand (Ed.), Science inquiry, argument and language: A case for the science writing heuristic. Sense Publishers.
35. Hand, B., & Keys, C. (1999). Inquiry investigation: A new approach to laboratory reports. The Science Teacher, 66(1), 27-29.
36. Hand, B., & Norton Meier, L. (Eds.) (2011). Voices from the classroom: Elementary teachers’ experience with argument based inquiry. Sense Publishers.
37. Hand, B., Wallace, C., & Yang, E. (2004). Using the science writing heuristic to enhance learning outcomes from laboratory activities in seventh grade science: Quantitative and qualitative aspects. International Journal of Science Education, 26(1), 131 149. https://doi.org/10.1080/0950069032000070252
38. İsbir, B., & Yıldız, A. (2021). Argümantasyon yönteminin uygulanması sürecinde karşılaşılan sınırlılıkların tartışılması [Discussing the limitations during the implementation of the argumentation method]. Journal of Social Research and Behavioral Sciences, 7(13), 236 258. https://doi.org/10.52096/jsrbs.6.1.7.13.13
39. Jimenez Aleixandre, M., & Erduran, S. (2008). Argumentation in science education: An Overview. In M Jimenez Aleixandre & S. Erduran (Eds.), Argumentation in science education: Perspectives from classroom-based research (pp. 3-27). Springer
40. Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130 144. https://doi.org/10.1016/j.edurev.2007.05.002
41. Kabataş Memiş, E. (2017). Argümantasyon uygulamalarına katılan öğretmen adaylarının küçük grup tartışmalarına ilişkin görüşleri [Opinions of teacher candidate on small group discussions in argumentation applications]. Kastamonu Education Journal, 25(5), 2037 2056. Retrieved from https://dergipark.org.tr/en/pub/kefdergi/issue/31226/342940
42. Kaya, O.N, & Kılıç, Z. (2008). Etkin bir fen eğitim için tartışmacı söylev [Argumentative discourse for the effective teaching of science]. Ahi Evran University Journal of Kırşehir Education Faculty, 9(3), 89 100. https://dergipark.org.tr/en/pub/kefad/issue/59524/855999
43. King, P.M. (2000). Learning to make reflective judgments. In M.B. Baxter Magolda (Ed.), Linking student development, learning, and teaching: New directions for teaching and learning (pp. 15–26). Jossey-Bass.
44. Kutlu, Ö., Doğan, C.D., & Karakaya, İ. (2009). Öğrenci başarısının belirlenmesi performansa ve portfolyoya dayalı durum belirleme [Determining student success, determining the situation based on performance and portfolio] (2nd ed.). Pegem Akademi.
45. Kılıç, S. (2015). Kappa test. Journal of Mood Disorders, 5(3), 142 144. https://doi.org/10.5455/jmood.20150920115439
46. Landis, J.R., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159 174. https://doi.org/10.2307/2529310
47. Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel psychology, 28(4), 563 575.
48. Leou, M., Abder, P., Riordan, M., & Zoller, U. (2006). ‘Using HOCS centered learning’ as a pathway to promote science teachers’ metacognitive development. Research in Science Education, 36(1 2), 69 84. https://doi.org/10.1007/s11165-005-3916-9
49. Martin, A.M., & Hand, B. (2007). Factors affecting the implementation of argument in the elementary science classroom. A longitudinal case study. Research in Science Education, 39, 17 38. https://doi.org/10.1007/s11165-007-9072-7
50. McGartland, R.D., Berg Weger, M., Tebb, S., Lee, E.S., & Rauch, S. (2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94 104. https://doi.org/10.1093/swr/27.2.94
51. McNeill, K.L., Lizotte, D., Krajcik, J.S., & Marx, R.W. (2006). Fading scaffolds for argumentation and explanation. Journal of the Learning Sciences, 15(2), 153 191. https://doi.org/10.1207/s15327809jls1502_1
52. McNeill, K.L., Gonzalez Howard, M., Katsh Singer, R., & Loper, S. (2017). Moving beyond pseudoargumentation: Teachers’ enactments of an educative science curriculum focused on argumentation. Science Education, 101(3), 426 457. https://doi.org/10.1002/sce.21274
53. Ministry of National Education (2018). İlköğretim kurumları fen bilimleri dersi öğretim programı [Primary education institutions’ science instruction program]. Talim Terbiye Kurulu Başkanlığı. http://mufredat.meb.gov.tr/Dosyalar/201812312311937
54. Moskal, B.M., & Leydens, J.A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research, and Evaluation, 7(1), 1 10. https://doi.org/10.7275/q7rm-gg74
55. Namdar, B., & Tuskan, İ.B. (2018). Fen bilgisi öğretmenlerinin argümantasyona yönelik görüşleri [Science teachers’ views of scientific argumentation]. Hacettepe University Journal of Education, 33(1), 1-22. https://doi.org/10.16986/HUJE.2017030137
56. Nam, J., Choi, A., & Hand, B. (2011). Implementation of the science writing heuristic (SWH) approach in 8th grade science classrooms. International Journal of Science and Mathematics Education, 9(5), 1111 1133. https://doi.org/10.1007/s10763-010-9250-3
57. National Research Council [NRC] (2000). Inquiry and the national science education standards. National Academic Press.
58. National Research Council. (2011). Successful STEM education: A workshop summary. National Academies Press.
59. National Science Teaching Association [NSTA] (2020). Nature of Science. https://www.nsta.org/nstas-official-positions/nature-science
60. NGSS Lead States (2013). Next generation science standards: For states, by states. National Academies. http://www.nextgenscience.org/next-generation-science-standards
61. Nam, J., Choi, A., & Hand, B. (2011). Implementation of the science writing heuristic (SWH) approach in 8th grade science classrooms. International Journal of Science and Mathematics Education, 9(1), 1111-1133. https://doi.org/10.1007/s10763-010-9250-3
62. Osborne, J. (2005). The role of argument in science education. Research and the Quality of Science Education, 7(1), 367-380. https://doi.org/10.1007/1-4020-3673-6_29
63. Öngöz, S. (2011). Elektronik ders kitabı değerlendirme formunun geliştirilmesi: Geçerlik ve güvenirlik çalışması [Development of electronic textbook evaluation form: Validity and reliability study]. 11th International Educational Technology Conference, IETC: Proceedings Book (Volume II), pp.1481-1485.
64. Sampson, V., & Blanchard, M.R. (2012). Science teachers and scientific argumentation: Trends in views and practice. Journal of Research in Science Teaching, 49(9), 1122 1148. http://doi.org/10.1002/tea.21037
65. Sandoval, W.A., & Millwood, K. (2005). The quality of students’ use of evidence in written scientific explanations. Cognition and Instruction, 23(1), 23 55. https://doi.org/10.1207/s1532690xci2301_2
66. Sevgi, Y., & Şahin, F. (2017). The effects of discussion the socio scientific subject in the newspaper based on argumentation 7th grades students’ critical thinking. Journal of Human Sciences, 14(1), 156 170. https://j humansciences.com/ojs/index.php/IJHS/article/view/4289
67. Sampson, V., & Blanchard, M. (2012). Science teachers and scientific argumentation: Trends in views and practice. Journal of Research in Science Teaching, 49(9), 1122 1148. https://doi.org/10.1002/tea.21037
68. Simon, S., Erduran, S., & Osborne, J. (2006). Learning to teach argumentation: Research and development in the science classroom. International Journal of Science Education, 28(23), 235 260. http://doi.org/10.1080/09500690500336957
69. Simon, S., & Johnson, S. (2008). Professional learning portfolios for argumentation in school science. International Journal of Science Education, 30(5), 669 688. https://doi.org/10.1080/09500690701854873
70. Şahin, A., Ayar, M.C., & Adiguzel, T. (2014). STEM related after school program activities and associated outcomes on student learning. Educational Sciences: Theory and Practice, 14(1), 309-322. https://doi.org/10.12738/estp.2014.1.1876
71. Şencan, H. (2005). Sosyal ve davranışsal ölçmelerde güvenirlik ve geçerlilik [Reliability and validity in social and behavioral measures]. SeckinYayıncılık.
72. Stohlmann, M., Moore, T.J., & Roehrig, G.H. (2012). Considerations for teaching integrated STEM education. Journal of Pre college Engineering Education Research (J PEER), 2(1), 28-34. https://doi.org/10.5703/1288284314653
73. Toulmin, S. (1958). The uses of argument. Cambridge University Press.
74. Torun, F., & Şahin, S. (2016). Determination of students' argument levels in argumentation based social studies course. Education and Science, 41(186), 233 251. https://doi.org/10.15390/EB.2016.6322
75. Tunkham, P., Donpudsa, S., & Dornbundit, P. (2016). Development of STEM activities in chemistry on “protein” to enhance 21st century learning skills for senior high school students. Silpakorn University Journal of Social Sciences, Humanities, and Arts, 16(3), 217 234. https://doi.org/10.14456/sujsha.2016.17
76. Turkish Industry and Business Association [TÜSİAD] (2017). Faaliyet raporu [Activity report]. https://tusiad.org/tr/faaliyet-raporlari/item/9911-tusiad-faaliyet-raporu-2017
77. Türkmenoğlu, M., & Çopur, E. (2021). Sınıf öğretmenlerinin argümantasyona ilişkin görüşlerinin ve argüman oluşturma düzeylerinin incelenmesi. Uluslararası Temel Eğitim Çalışmaları Dergisi, 2(1), 29 42. https://dergipark.org.tr/en/pub/ijpes/issue/60113/777604
78. Wilson, F.R., Pan, W., & Schumsky, D.A. (2012). Recalculation of the critical values for Lawshe’s content validity ratio. Measurement and Evaluation in Counseling and Development, 45(1), 197 210. https://doi.org/10.1177/0748175612440286
79. Wolf, K., & Stevens, E. (2007). The role of rubrics in advancing and assessing student learning. The Journal of Effective Teaching, 7(1), 3-14.
80. Yesildag Hasancebi, F., & Günel, M. (2014). Delving into the effect of argumentation based inquiry approach on learning science from multiple perspectives. Journal of Research in Education and Society, 1(1), 23-44.
81. Yeşilyurt, S., & Çapraz, C. (2018). A road map for the content validity used in scale development studies. Erzincan Üniversitesi Eğitim Fakültesi Dergisi, 20(1), 251 264. https://doi.org/10.17556/erziefd.297741
82. Yaghmaie, F. (2003). Content validity and its estimation. Journal of Medical Education, 3(1), 25-27.
83. Yıldırır, H.E., & Nakiboğlu, C. (2014). Kimya öğretmen ve öğretmen adaylarının derslerinde kullandıkları argümantasyon süreçlerinin incelenmesi [Examination of chemistry teachers and preservice teachers’argumentation processes used in their courses]. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 14(2), 124 154. https://hdl.handle.net/20.500.12462/4228
84. Yurdugül, H. (2005). Ölçek geliştirme çalışmalarında kapsam geçerliği için kapsam geçerlik indekslerinin kullanılması [Using content validity indexes for content validity in scale development studies]. XIV. Ulusal Eğitim Bilimleri Kongresi [XIV. National Educational Sciences Congress], 1, 771-774.
85. Zembal Saul, C., & Vaishampayan, A. (2019). Science teachers’ continuous professional development inargumentation. In S. Erduran (Ed.), Argumentation in chemistry education: Research, policy and practice (pp. 142 172). Royal Society of Chemistry.
86. Zohar, A. (2007). Science teacher education and professional development in argumentation. In S. Erduran & M. Jiménez-Aleixandre (Eds.), Argumentation in science education (pp. 245–268). Springer.