Development and Validation of a Two-Tier Diagnostic Test to Assess Seventh-Grade Students’ Conceptual Understanding of Pure Substances and Mixtures

Abstract

This study aimed to develop and validate a two-tier diagnostic concept test designed to assess seventh-grade students’ conceptual understanding and reasoning about this topic. The first tier consisted of multiple-choice items, whereas the second tier required students to provide open-ended explanations for their selected answers. The study was conducted with 200 seventh-grade students from five middle schools. An initial 25-item instrument was developed based on curriculum outcomes, misconceptions reported in the literature, expert review, and pilot testing. Student responses were analysed using Classical Test Theory and Rasch modelling. Item analysis identified seven items with weaker psychometric performance, resulting in a final 18-item version. The final form demonstrated acceptable item difficulty and discrimination indices, high internal consistency (Cronbach’s alpha = .879), satisfactory person and item reliability, acceptable Rasch fit statistics, and appropriate targeting between item difficulty and student ability. The findings provide evidence supporting the instrument's use as a diagnostic tool for identifying students’ conceptual understanding, reasoning patterns, and misconceptions regarding pure substances and mixtures.

Keywords

• Two-tier diagnostic test
• Conceptual Learning
• Pure Substances and Mixtures
• Rasch analysis

Supporting Institution

Erciyes University Scientific Research Projects Unit

Project Number

SYL-2022-11698

Ethical Statement

We hereby declare that the study has not unethical issues and that research and publication ethics have been observed carefully. Ethics committee approval was obtained for this study from the Erciyes University, Scientific Research and Publication Ethics Committee, with its decision dated 25/01/2022 and numbered 38.

Thanks

This research was supported by the Erciyes University Scientific Research Projects Unit under project code SYL-2022-11698. The authors would like to thank the Erciyes University Scientific Research Projects Unit for their financial and institutional support.

Yedinci Sınıf Öğrencilerinin Saf Maddeler ve Karışımlar Konusundaki Kavramsal Anlayışlarını Değerlendirmeye Yönelik İki Aşamalı Tanılayıcı Testin Geliştirilmesi ve Geçerliğinin Sağlanması

Abstract

Bu çalışmanın amacı, yedinci sınıf öğrencilerinin bu konuya ilişkin kavramsal anlayışlarını ve akıl yürütme süreçlerini değerlendirmek üzere tasarlanmış iki aşamalı tanılayıcı bir kavram testi geliştirmek ve geçerliğini sağlamaktır. Testin birinci aşaması çoktan seçmeli maddelerden oluşurken, ikinci aşamada öğrencilerden seçtikleri yanıtlara ilişkin açık uçlu açıklamalar yapmaları istenmiştir. Çalışma, beş ortaokuldan 200 yedinci sınıf öğrencisi ile yürütülmüştür. Başlangıçta 25 maddelik bir ölçme aracı; öğretim programı kazanımları, alanyazında bildirilen kavram yanılgıları, uzman görüşü ve pilot uygulama temel alınarak geliştirilmiştir. Öğrenci yanıtları Klasik Test Kuramı ve Rasch modellemesi kullanılarak analiz edilmiştir. Madde analizi, psikometrik performansı daha zayıf olan yedi maddeyi belirlemiş ve bunun sonucunda nihai test 18 maddelik bir forma dönüştürülmüştür. Nihai form; kabul edilebilir madde güçlük ve ayırt edicilik indeksleri, yüksek iç tutarlılık düzeyi (Cronbach alfa = .879), tatmin edici kişi ve madde güvenirliği, kabul edilebilir Rasch uyum istatistikleri ve madde güçlüğü ile öğrenci yeteneği arasında uygun hedefleme göstermiştir. Bulgular, bu ölçme aracının öğrencilerin saf maddeler ve karışımlar konusundaki kavramsal anlayışlarını, akıl yürütme örüntülerini ve kavram yanılgılarını belirlemeye yönelik tanılayıcı bir araç olarak kullanılmasını destekleyen kanıtlar sunmaktadır.

Keywords

• İki aşamalı tanı testi
• Kavramsal Öğrenme
• Saf Maddeler ve Karışımlar
• Rasch Analizi

Project Number

SYL-2022-11698

References

1. Altay, E., & Balım, A. G. (2021). Maddenin tanecikli yapısı ile ilgili kavram yanılgılarının tespiti. Batı Anadolu Eğitim Bilimleri Dergisi, 12(2), 576–592.
2. Ayas, A., Özmen, H., & Çalık, M. (2010). Students’ conceptions of the PNM at secondary and tertiary level. International Journal of Science and Mathematics Education, 8, 165–184.
3. Barke, H. D., Harsch, G., & Krerecke, S. (2012). Essentials of chemical education. Springer Nature.
4. Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.
5. Boz, Y. (2006). Turkish pupils’ conceptions of the particulate nature of matter. Journal of Science Education and Technology, 15(2), 203–213.
6. Bridle, C. A., & Yezierski, E. J. (2012). Evidence for the effectiveness of inquiry-based, particulate-level instruction on conceptions of the PNM. Journal of Chemical Education, 89, 192–198.
7. Broman, K., & Simon, S. (2021). Upper secondary school students’ chemistry understanding: The case of pure substances and mixtures. Chemistry Education Research and Practice, 22(4), 1017–1030.
8. Bybee, R. W. (2013). The case for STEM education: Challenges and opportunities. NSTA Press.

9. Chandrasegaran, A. L., Treagust, D. F., & Mocerino, M. (2007). The development of a two-tier multiple-choice diagnostic instrument for evaluating secondary school students’ ability to describe and explain chemical reactions using multiple levels of representation. Chemistry Education Research and Practice, 8(3), 293–307. https://doi.org/10.1039/B7RP90006F
10. Çalık, M., Ayas, A., & Ebenezer, J. V. (2006). A synthesis of a series of studies of solution chemistry with respect to the Greek model of mind. Journal of Science Education and Technology, 15(5), 417–434.
11. Chi, M. T. H. (2008). Three types of conceptual change: Belief revision, mental model transformation, and categorical shift. In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. 61–82). Routledge.
12. DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). SAGE Publications.
13. Driver, R., Squires, A., Rushworth, P., & Wood-Robinson, V. (1994). Making sense of secondary science: Research into children’s ideas. Routledge.
14. Duit, R., & Treagust, D. F. (2012). How can conceptual change contribute to theory and practice in science education? In Second international handbook of science education (pp. 31–46). Springer.
15. Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Prentice Hall.
16. Ebenezer, J. V., & Erickson, G. L. (1996). Chemistry students’ conceptions of solubility: A phenomenography. Science Education, 80(2), 181–201.
17. Griffiths, A. K., & Preston, K. R. (1992). Grade-12 students’ misconceptions relating to fundamental characteristics of atoms and molecules. Journal of Research in Science Teaching, 29(6), 611–628.
18. Gkitzia, V., Salta, K., & Tzougraki, C. (2020). Students’ competence in translating between different types of chemical representations. Chemistry Education Research and Practice, 21(1), 307–330. https://doi.org/10.1039/C8RP00301G
19. Gurel, D. K., Eryilmaz, A., & McDermott, L. C. (2015). A review and comparison of diagnostic instruments to identify students’ misconceptions in science. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 989–1008.
20. Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333.
21. Kind, V. (2004). Beyond appearances: Students’ misconceptions about basic chemical ideas. Royal Society of Chemistry.
22. Lee, O., Eichinger, D. C., Anderson, C. W., Berkheimer, G. D., & Blakeslee, T. D. (1993). Changing middle school students’ conceptions of matter and molecules. Journal of Research in Science Teaching, 30(3), 249–270.
23. Luviani, S. D., Mulyani, S., & Widhiyanti, T. (2021). A review of three levels of chemical representation until 2020. Journal of Physics: Conference Series, 1806(1), Article 012206. https://doi.org/10.1088/1742-6596/1806/1/012206 Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174.
24. Nakhleh, M. B., Samarapungavan, A., & Sağlam, Y. (2005). Middle school students’ beliefs about matter. Journal of Research in Science Teaching, 42(5), 581–612.
25. National Research Council. (2012). A framework for K–12 science education: Practices, crosscutting concepts, and core ideas. National Academies Press.
26. Nunnally, J. C., & Bernstein, I. R. (1994). Psychometric theory. McGraw-Hill.
27. OECD. (2019). PISA 2018 assessment and analytical framework. OECD Publishing.
28. Özmen, H. (2011). Effect of animation enhanced conceptual change texts on the 6th grade students’ understanding of the particulate nature of matter and transformation during phase changes. Computers & Education, 57, 1114–1126.
29. Øyehaug, A. B., & Holt, A. (2013). Students’ understanding of the nature of matter and chemical reactions: A longitudinal study of conceptual restructuring. Chemistry Education Research and Practice, 14, 450–467.
30. Posner, G. J., Strike, K. A., Hewson, P. W., & Gertzog, W. A. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science Education, 66(2), 211–227.
31. Radzuan, N. S. B. M., & Hanri, C. B. (2024). Science students’ misconceptions of basic concepts of chemistry on matter and its particulate properties. International Journal of Academic Research in Progressive Education and Development, 13(4), 2351–2360. https://doi.org/10.6007/IJARPED/v13-i4/23824
32. Sharma, S., & Kotecha, K. (2024). Analysis of conceptual understanding of solutions and titration from a submicroscopic perspective. Cogent Education, 11(1), 1–24. https://doi.org/10.1080/2331186X.2024.2315834
33. Soeharto, S., Csapó, B., Sarimanah, E., Dewi, F. I., & Sabri, T. (2021). A review of students’ common misconceptions in science and their diagnostic assessment tools. Jurnal Pendidikan IPA Indonesia, 10(2), 248–259.
34. Taber, K. S. (2002). Chemical misconceptions: Prevention, diagnosis and cure (Vol. 1). Royal Society of Chemistry.
35. Taber, K. S. (2009). Progressing science education: Constructing the scientific research programme into the contingent nature of learning. Springer.
36. Taber, K. S. (2013). Revisiting the chemistry triplet: Drawing upon the nature of chemical knowledge and the psychology of learning to inform chemistry education. Chemistry Education Research and Practice, 14(2), 156–168. https://doi.org/10.1039/C3RP00012E
37. Taber, K. S. (2019). The nature of the chemical concept: Re-constructing chemical knowledge in teaching and learning. Royal Society of Chemistry.
38. Talanquer, V. (2013). School chemistry: The need for transgression. Science & Education, 22(7), 1757–1773. Talanquer, V. (2020). Chemical thinking: A review of its characteristics and development. Journal of Chemical Education, 97(12), 4272–4283.
39. Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students’ misconceptions in science. International Journal of Science Education, 10(2), 159–169.
40. Vosniadou, S. (2013). Conceptual change in learning and instruction: The framework theory approach. In S. Vosniadou (Ed.), The international handbook of conceptual change (2nd ed., pp. 11–30). Routledge.
41. Vosniadou, S. (2019). The development of students’ understanding of science. Frontiers in Education, 4, Article 32. https://doi.org/10.3389/feduc.2019.00032
42. Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.