Revising of the Integrating Scientific Literacy Skills Scale (ISLS) with Rasch Model Analysis

Abstract

In this study it was aimed to develop and analyze instruments of integrating scientific literacy skills scale (ISLS) for science program students of senior high school with a Rasch Model Analysis. In developing and analyzing instruments we use the Messick’s validity (1996) approach which consists of five aspects including content, substantive, structural, external, and consequential. ISLS consisted of 14 cases of integrated science presented in the form of a testlet. Each case consists of three questions given to scientific literacy competencies according to PISA 2015 standards. The research design uses the ADDIE procedural model (Analysis, Design, Development, Implementation, Evaluation). Participants consisted of 310 grade XII students of the science program from two senior high schools in Tegal City, Indonesia. Constructive validation with Rasch modelling gives the following results. The level of conformity of the items is in the range of -3 to 4. All the items that are suitable for modelling. As many as 95.16 % of student responses match modelling. Has no items containing DIF. It can be said that ISLS, which consists of 14 items, is suitable for measuring Integrating Scientific Literacy Skills for science program students of senior high school.

Keywords

• Rasch model
• Integrating Scientific Literacy Skills Scale (ISLS)
• Scale development
• Science program students

References

1. Abdul Rahim, F., & Chun, L. S. (2017). Proposing an affective literacy framework for young learners of English in Malaysian rural areas: Its key dimensions and challenges. Malaysian Journal of Learning and Instruction, 14(2), 115–144. https://doi.org/10.32890/mjli2017.14.2.5
2. Adeleke, A. A., & Joshua, E. O. (2015). Development and Validation of Scientific Literacy Achievement Test to Assess Senior Secondary School Students ’ Literacy Acquisition in Physics. Journal of Education and Practice, 6(7), 28–43.
3. Babcock, B., & Albano, A. D. (2012). Rasch scale stability in the presence of item parameter and trait drift. Applied Psychological Measurement, 36(7), 565-580.
4. Baghaei, P., & Amrahi, N. (2011). Validation of a Multiple Choice English Vocabulary Test with the Rasch Model. Journal of Language Teaching and Research, 2(5), 1052-1060. https://doi.org/10.4304/jltr.2.5.1052-1060
5. Bates, S., Donnelly, R., Macphee, C., Sands, D., Birch, M., & Walet, N. R. (2013). Gender differences in conceptual understanding of Newtonian mechanics: A UK cross-institution comparison. European Journal of Physics, 34(2), 421–434. https://doi.org/10.1088/0143-0807/34/2/421
6. Bond, T., Yan, Z., & Heene, M. (2020). Applying the Rasch model: Fundamental measurement in the human sciences. Routledge.
7. Branch, R. M. (2009). Instructional design: The ADDIE approach (Vol. 722). Springer Science & Business Media.
8. Benjamin, T.E., Marks, B., Demetrikopoulos, M.K., Rose, J., Pollard, E., Thomas, A., & Muldrow, L.L. (2017). Development and Validation of Scientific Literacy Scale for College Preparedness in STEM with Freshmen from Diverse Institutions. International Journal of Science and Mathematics Education, 15, 607–623. https://doi.org/10.1007/s10763-015-9710-x

9. Bybee, R. W. (2012). Scientific literacy in environmental and health education. In Science / Environment / Health: Towards a Renewed Pedagogy for Science Education (Vol. 9789048139491, pp. 49–67). Springer Netherlands. https://doi.org/10.1007/978-90-481-3949-1_4
10. Chen, W. H., Lenderking, W., Jin, Y., Wyrwich, K. W., Gelhorn, H., & Revicki, D. A. (2014). Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Quality of life research, 23(2), 485-493.
11. Dietz, R. D., Pearson, R. H., Semak, M. R., & Willis, C. W. (2012). Gender bias in the force concept inventory? AIP Conference Proceedings, 1413, 171–174. https://doi.org/10.1063/1.3680022
12. Edwards, A., & Alcock, A. (2010). Using rasch analysis to identify uncharacteristic responses to undergraduate assessments. Teaching Mathematics and Its Applications, 29(4), 165–175. https://doi.org/10.1093/teamat/hrq008
13. Gormally, C., Brickman, P., & Lut, M. (2012). Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates’ evaluation of scientific information and arguments. CBE Life Sciences Education, 11(4), 364–377. https://doi.org/10.1187/cbe.12-03-0026
14. Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.
15. Hanson, S. (2016). The assessment of scientific reasoning skills of high school science students: A standardized assessment instrument. http://ir.library.illinoisstate.edu/cgi/viewcontent.cgi?article=1505&context=etd
16. Hanushek, E. A., & Woessmann, L. (2016). Knowledge capital, growth, and the East Asian miracle. Science, 351(6271), 344–345. https://doi.org/10.1126/science.aad7796
17. Heng, L. L., Surif, J., Seng, C. H., & Ibrahim, N. H. (2015). Mastery of scientific argumentation on the concept of neutralization in chemistry: A Malaysian perspective. Malaysian Journal of Learning and Instruction, 12(1), 85–101. https://doi.org/10.32890/mjli2015.12.5
18. Herrmann-Abell, C. F., & DeBoer, G. E. (2011). Using distractor-driven standards-based multiple-choice assessments and Rasch modeling to investigate hierarchies of chemistry misconceptions and detect structural problems with individual items. Chemistry Education Research and Practice, 12(2), 184–192. https://doi.org/10.1039/c1rp90023d
19. Hohensinn, C., & Kubinger, K. D. (2011). On the impact of missing values on item fit and the model validness of the Rasch model. Psychological Test and Assessment Modeling, 53(3), 380–393. http://www.psychologie-aktuell.com/fileadmin/download/ptam/3-2011_20110927/07_Hohensinn.pdf
20. Holbrook, J., & Rannikmae, M. (2009). The Meaning of Scientific Literacy. In ERIC. http://www.ijese.com/
21. Jufri, A. W., Hakim, A., & Ramdani, A. (2019). Instrument Development in Measuring the Scientific Literacy Integrated Character Level of Junior High School Students. Journal of Physics: Conference Series, 1233(1). https://doi.org/10.1088/1742-6596/1233/1/012100
22. Lamb, R. L., Annetta, L., Meldrum, J., & Vallett, D. (2012). Measuring Science Interest: Rasch Validation of the Science Interest Survey. International Journal of Science and Mathematics Education, 10(3), 643–668. https://doi.org/10.1007/s10763-011-9314-z
23. Lamprianou, I. (2010). The practical application of Optimal Appropriateness Measurement on empirical data using rasch models. Journal of Applied Measurement, 11(4), 409–423.
24. Liu, M. T., & Yu, P. T. (2011). Aberrant learning achievement detection based on person-fit statistics in personalized e-learning systems. Educational Technology and Society, 14(1), 107–120.
25. Madsen, A., McKagan, S. B., & Sayre, E. C. (2013). Gender gap on concept inventories in physics: What is consistent, what is inconsistent, and what factors influence the gap? Physical Review Special Topics-Physics Education Research, 9(2), 020121-020136. https://doi.org/10.1103/PhysRevSTPER.9.020121
26. Magis, D., Raîche, G., & Béland, S. (2012). A didactic presentation of snijders’s l z* index of person fit with emphasis on response model selection and ability estimation. In Journal of Educational and Behavioral Statistics, 37(1), 57–81. https://doi.org/10.3102/1076998610396894
27. Mari, L., Carbone, P., & Petri, D. (2012). Measurement fundamentals: A pragmatic view. IEEE Transactions on Instrumentation and Measurement, 61(8), 2107–2115. https://doi.org/10.1109/TIM.2012.2193693
28. Md-Ali, R., Karim, H. B. B. A., & Yusof, F. M. (2016). Experienced primary school teachers’ thoughts on effective teachers of literacy and numeracy. Malaysian Journal of Learning and Instruction, 13(1), 43–62. https://doi.org/10.32890/mjli2016.13.1.3
29. Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241–256. https://doi.org/10.1177/026553229601300302
30. Meyer, J. ., & Zhu, S. (2013). Fair and Equitable Measurement of Student Learning in MOOCs: An Introduction to Item Response Theory, Scale Linking, and Score Equating. Research & Practice in Assessment, 8, 26–39. http://www.rpajournal.com/dev/wp-content/uploads/2013/05/SF3.pdf&sa=X&scisig=AAGBfm2n7WN_mLyfzwk1qYsjHvIek10RhA&oi=scholarr&ei=JU2iUtXYMJGrhQfYz4HQCA&ved=0CDAQgAMoATAA
31. Morris, G. A., Harshman, N., Branum-Martin, L., Mazur, E., Mzoughi, T., & Baker, S. D. (2012). An item response curves analysis of the Force Concept Inventory. American Journal of Physics, 80(9), 825–831. https://doi.org/10.1119/1.4731618
32. Neumann, I., Neumann, K., & Nehm, R. (2010). Evaluating instrument quality in science education: Rasch-based analyses of a Nature of Science Test. Taylor & Francis. https://doi.org/10.1080/09500693.2010.511297ï
33. Nordin, H., & Ariffin, T. F. T. (2016). Validation of a technological pedagogical content knowledge instrument in a Malaysian secondary school context. Malaysian Journal of Learning and Instruction, 13(1), 1–24. https://doi.org/10.32890/mjli2016.13.1.1
34. OECD. (2015). OECD iLibrary | PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy. https://www.oecd-ilibrary.org/education/pisa-2015-assessment-and-analytical-framework_9789264255425-en
35. Planinic, M., Ivanjek, L., & Susac, A. (2010). Rasch model based analysis of the Force Concept Inventory. Physical Review Special Topics - Physics Education Research, 6(1). https://doi.org/10.1103/PhysRevSTPER.6.010103
36. Pratiwi, M., Siahaan, P., Samsudin, A., Aminudin, A. H., & Rachmadtullah, R. (2020). Introduction , Connection , Application , Reflection , Extension-Multimedia Based Integrated Instruction ( ICARE-U ): A Model to Improve Creative Thinking Skills. 24(08).
37. Ravand, H., & Firoozi, T. (2016). Examining construct validity of the master’s UEE using the Rasch model and the six aspects of the Messick's framework. International Journal of Language Testing, 6(1), 1-18.
38. Romine, W. L., Schaffer, D. L., & Barrow, L. (2015). International Journal of Science Education Development and Application of a Novel Rasch-based Methodology for Evaluating Multi-Tiered Assessment Instruments: Validation and utilization of an undergraduate diagnostic test of the water cycle. Taylor & Francis, 37(16), 2740–2768. https://doi.org/10.1080/09500693.2015.1105398
39. Rudolph, J. L., & Horibe, S. (2016). What do we mean by science education for civic engagement? Journal of Research in Science Teaching, 53(6), 805–820. https://doi.org/10.1002/tea.21303
40. Runnels, J. (2012). Using the Rasch model to validate a multiple choice English achievement test. International Journal of Language Studies, 6(4), 141-155.
41. Rusilowati, A., Kurniawati, L., Nugroho, S. E., & Widiyatmoko, A. (2016). Developing an instrument of scientific literacy asessment on the cycle theme. International Journal of Environmental and Science Education, 11(12), 5718–5727.
42. Rusilowati, A., Nugroho, S. E., Susilowati, E. S. M., Mustika, T., Harfiyani, N., & Prabowo, H. T. (2018). The development of scientific literacy assessment to measure student’s scientific literacy skills in energy theme. Journal of Physics: Conference Series, 983(1). https://doi.org/10.1088/1742-6596/983/1/012046
43. Saddhono, K., & Rohmadi, M. (2014). A sociolinguistics study on the use of the Javanese language in the learning process in primary schools in Surakarta, Central Java, Indonesia. International Education Studies, 7(6), 25–30. https://doi.org/10.5539/ies.v7n6p25
44. Samsudin, A. (2020). Rasch Analysis: Measuring Students Attitudes toward Physics using the CLASS. Test Engineering & Management, 83(June), 15461–15467.
45. Sheu, T. W., Tsai, C. P., Tzeng, J. W., Chen, T. L., & Nagai, M. (2013). An algorithm of the misconception order. Applied Mechanics and Materials, 284-287, 3010–3014. https://doi.org/10.4028/www.scientific.net/AMM.284-287.3010
46. Shu, Z., Henson, R., & Luecht, R. (2013). Using Deterministic, Gated Item Response Theory Model to Detect Test Cheating due to Item Compromise. Psychometrika, 78(3), 481–497. https://doi.org/10.1007/s11336-012-9311-3
47. Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26(1), 100-107.
48. Smith, A. B., Fallowfield, L. J., Stark, D. P., Velikova, G., & Jenkins, V. (2010). A Rasch and confirmatory factor analysis of the General Health Questionnaire (GHQ) - 12. Health and Quality of Life Outcomes, 8. https://doi.org/10.1186/1477-7525-8-45
49. Soobard, R., & Rannikmäe, M. (2011). Assessing student’s level of scientific literacy using interdisciplinary scenarios. In Science Education International, 22(2), 133-144. https://eric.ed.gov/?id=EJ941672
50. Stenbeck, M., Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1992). Fundamentals of Item Response Theory. Contemporary Sociology, 21(2), 289. https://doi.org/10.2307/2075521
51. Suhandi, A., & Samsudin, A. (2019). Effectiveness of the use of developed teacher’s book in guiding the implementation of physics teaching that provides science literacy and instill spiritual attitudes. Journal of Physics: Conference Series, 1280(5). https://doi.org/10.1088/1742-6596/1280/5/052054
52. Sumintono, B. (2018). Rasch Model Measurements as Tools in Assesment for Learning. https://doi.org/10.2991/icei-17.2018.11
53. Suryana, T. G. S., Setyadin, A. H., Samsudin, A., & Kaniawati, I. (2020). Assessing Multidimensional Energy Literacy of High School Students: An Analysis of Rasch Model. Journal of Physics: Conference Series, 1467(1), 1-10. https://doi.org/10.1088/1742-6596/1467/1/012034
54. Susongko, P. (2016). Validation of science achievement test with the Rasch model. Jurnal Pendidikan IPA Indonesia, 5(2), 268–277. https://doi.org/10.15294/jpii.v5i2.7690
55. Tamassia, L., & Frans, R. (2014). Does integrated science education improve scientific literacy? In Journal of the European Teacher Education Network, 9(1),131–141. http://62.28.241.73/index.php/jeten/article/view/44
56. Wagner-Menghin, M., Preusche, I., & Schmidts, M. (2013). The Effects of Reusing Written Test Items: A Study Using the Rasch Model. ISRN Education, 2013, 1–7. https://doi.org/10.1155/2013/585420
57. Wilson, K., Low, D., Verdon, M., & Verdon, A. (2016). Differences in gender performance on competitive physics selection tests. Physical Review Physics Education Research, 12(2), 020111. https://doi.org/10.1103/PhysRevPhysEducRes.12.020111
58. Wind, S. A., & Gale, J. D. (2015). Diagnostic Opportunities Using Rasch Measurement in the Context of a Misconceptions-Based Physical Science Assessment. Science Education, 99(4), 721–741. https://doi.org/10.1002/sce.21172
59. Winne, P. H. (2020). Construct and consequential validity for learning analytics based on trace data. Computers in Human Behavior, 112, 106457.
60. Wright, B. D., & Stone, M. H. (1979). Best Test Design: Rasch Measurement. Chicago, Illinois: MESA Press
61. Wright, B., & Mok, M. M. C. (2004). An Overview of the Family of Rasch Measurement Models. Introduction to Rasch Measurement, 1–24. http://www.statistica.unimib.it/utenti/lovaglio/overview rasch.pdf
62. Wu, M., & Adams, R. (2007). Applyıng the Rasch model to psycho-socıal measurement a Practıcal Approach https://pdfs.semanticscholar.org/a263/e16ffe74da6ec87d7855ed878d6ab90acfed.pdf
63. Yenni, R., Hernani, & Widodo, A. (2017). The implementation of integrated science teaching materials based socio-scientific issues to improve students scientific literacy for environmental pollution theme. AIP Conference Proceedings, 1848, 060002 https://doi.org/10.1063/1.4983970