Revising of the Integrating Scientific Literacy Skills Scale (ISLS) with Rasch Model Analysis

Year 2020, , 1583 - 1602, 15.12.2020

Purwo Susongko , Mobinta Kusuma Yuni Arfiani Achmad Samsudin , Adam Amınudın

https://doi.org/10.17478/jegys.781583

Cited By: 1

Abstract

In this study it was aimed to develop and analyze instruments of integrating scientific literacy skills scale (ISLS) for science program students of senior high school with a Rasch Model Analysis. In developing and analyzing instruments we use the Messick’s validity (1996) approach which consists of five aspects including content, substantive, structural, external, and consequential. ISLS consisted of 14 cases of integrated science presented in the form of a testlet. Each case consists of three questions given to scientific literacy competencies according to PISA 2015 standards. The research design uses the ADDIE procedural model (Analysis, Design, Development, Implementation, Evaluation). Participants consisted of 310 grade XII students of the science program from two senior high schools in Tegal City, Indonesia. Constructive validation with Rasch modelling gives the following results. The level of conformity of the items is in the range of -3 to 4. All the items that are suitable for modelling. As many as 95.16 % of student responses match modelling. Has no items containing DIF. It can be said that ISLS, which consists of 14 items, is suitable for measuring Integrating Scientific Literacy Skills for science program students of senior high school.

Keywords

Rasch model, Integrating Scientific Literacy Skills Scale (ISLS), Scale development, Science program students

Thanks

We would like to thank Kementerian Riset, Teknologi dan Pendidikan Tinggi Republic of Indonesia for providing funding for this research. Likewise, we would like to thank all parties involved, especially the Principals of SMA 2 and SMA 3 of Tegal City who has supported and granted research permits.

References

• Abdul Rahim, F., & Chun, L. S. (2017). Proposing an affective literacy framework for young learners of English in Malaysian rural areas: Its key dimensions and challenges. Malaysian Journal of Learning and Instruction, 14(2), 115–144. https://doi.org/10.32890/mjli2017.14.2.5
• Adeleke, A. A., & Joshua, E. O. (2015). Development and Validation of Scientific Literacy Achievement Test to Assess Senior Secondary School Students ’ Literacy Acquisition in Physics. Journal of Education and Practice, 6(7), 28–43.
• Babcock, B., & Albano, A. D. (2012). Rasch scale stability in the presence of item parameter and trait drift. Applied Psychological Measurement, 36(7), 565-580.
• Baghaei, P., & Amrahi, N. (2011). Validation of a Multiple Choice English Vocabulary Test with the Rasch Model. Journal of Language Teaching and Research, 2(5), 1052-1060. https://doi.org/10.4304/jltr.2.5.1052-1060
• Bates, S., Donnelly, R., Macphee, C., Sands, D., Birch, M., & Walet, N. R. (2013). Gender differences in conceptual understanding of Newtonian mechanics: A UK cross-institution comparison. European Journal of Physics, 34(2), 421–434. https://doi.org/10.1088/0143-0807/34/2/421
• Bond, T., Yan, Z., & Heene, M. (2020). Applying the Rasch model: Fundamental measurement in the human sciences. Routledge.
• Branch, R. M. (2009). Instructional design: The ADDIE approach (Vol. 722). Springer Science & Business Media.
• Benjamin, T.E., Marks, B., Demetrikopoulos, M.K., Rose, J., Pollard, E., Thomas, A., & Muldrow, L.L. (2017). Development and Validation of Scientific Literacy Scale for College Preparedness in STEM with Freshmen from Diverse Institutions. International Journal of Science and Mathematics Education, 15, 607–623. https://doi.org/10.1007/s10763-015-9710-x
• Bybee, R. W. (2012). Scientific literacy in environmental and health education. In Science / Environment / Health: Towards a Renewed Pedagogy for Science Education (Vol. 9789048139491, pp. 49–67). Springer Netherlands. https://doi.org/10.1007/978-90-481-3949-1_4
• Chen, W. H., Lenderking, W., Jin, Y., Wyrwich, K. W., Gelhorn, H., & Revicki, D. A. (2014). Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Quality of life research, 23(2), 485-493.
• Dietz, R. D., Pearson, R. H., Semak, M. R., & Willis, C. W. (2012). Gender bias in the force concept inventory? AIP Conference Proceedings, 1413, 171–174. https://doi.org/10.1063/1.3680022
• Edwards, A., & Alcock, A. (2010). Using rasch analysis to identify uncharacteristic responses to undergraduate assessments. Teaching Mathematics and Its Applications, 29(4), 165–175. https://doi.org/10.1093/teamat/hrq008
• Gormally, C., Brickman, P., & Lut, M. (2012). Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates’ evaluation of scientific information and arguments. CBE Life Sciences Education, 11(4), 364–377. https://doi.org/10.1187/cbe.12-03-0026
• Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.
• Hanson, S. (2016). The assessment of scientific reasoning skills of high school science students: A standardized assessment instrument. http://ir.library.illinoisstate.edu/cgi/viewcontent.cgi?article=1505&context=etd
• Hanushek, E. A., & Woessmann, L. (2016). Knowledge capital, growth, and the East Asian miracle. Science, 351(6271), 344–345. https://doi.org/10.1126/science.aad7796
• Heng, L. L., Surif, J., Seng, C. H., & Ibrahim, N. H. (2015). Mastery of scientific argumentation on the concept of neutralization in chemistry: A Malaysian perspective. Malaysian Journal of Learning and Instruction, 12(1), 85–101. https://doi.org/10.32890/mjli2015.12.5
• Herrmann-Abell, C. F., & DeBoer, G. E. (2011). Using distractor-driven standards-based multiple-choice assessments and Rasch modeling to investigate hierarchies of chemistry misconceptions and detect structural problems with individual items. Chemistry Education Research and Practice, 12(2), 184–192. https://doi.org/10.1039/c1rp90023d
• Hohensinn, C., & Kubinger, K. D. (2011). On the impact of missing values on item fit and the model validness of the Rasch model. Psychological Test and Assessment Modeling, 53(3), 380–393. http://www.psychologie-aktuell.com/fileadmin/download/ptam/3-2011_20110927/07_Hohensinn.pdf
• Holbrook, J., & Rannikmae, M. (2009). The Meaning of Scientific Literacy. In ERIC. http://www.ijese.com/
• Jufri, A. W., Hakim, A., & Ramdani, A. (2019). Instrument Development in Measuring the Scientific Literacy Integrated Character Level of Junior High School Students. Journal of Physics: Conference Series, 1233(1). https://doi.org/10.1088/1742-6596/1233/1/012100
• Lamb, R. L., Annetta, L., Meldrum, J., & Vallett, D. (2012). Measuring Science Interest: Rasch Validation of the Science Interest Survey. International Journal of Science and Mathematics Education, 10(3), 643–668. https://doi.org/10.1007/s10763-011-9314-z
• Lamprianou, I. (2010). The practical application of Optimal Appropriateness Measurement on empirical data using rasch models. Journal of Applied Measurement, 11(4), 409–423.
• Liu, M. T., & Yu, P. T. (2011). Aberrant learning achievement detection based on person-fit statistics in personalized e-learning systems. Educational Technology and Society, 14(1), 107–120.
• Madsen, A., McKagan, S. B., & Sayre, E. C. (2013). Gender gap on concept inventories in physics: What is consistent, what is inconsistent, and what factors influence the gap? Physical Review Special Topics-Physics Education Research, 9(2), 020121-020136. https://doi.org/10.1103/PhysRevSTPER.9.020121
• Magis, D., Raîche, G., & Béland, S. (2012). A didactic presentation of snijders’s l z* index of person fit with emphasis on response model selection and ability estimation. In Journal of Educational and Behavioral Statistics, 37(1), 57–81. https://doi.org/10.3102/1076998610396894
• Mari, L., Carbone, P., & Petri, D. (2012). Measurement fundamentals: A pragmatic view. IEEE Transactions on Instrumentation and Measurement, 61(8), 2107–2115. https://doi.org/10.1109/TIM.2012.2193693
• Md-Ali, R., Karim, H. B. B. A., & Yusof, F. M. (2016). Experienced primary school teachers’ thoughts on effective teachers of literacy and numeracy. Malaysian Journal of Learning and Instruction, 13(1), 43–62. https://doi.org/10.32890/mjli2016.13.1.3
• Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241–256. https://doi.org/10.1177/026553229601300302
• Meyer, J. ., & Zhu, S. (2013). Fair and Equitable Measurement of Student Learning in MOOCs: An Introduction to Item Response Theory, Scale Linking, and Score Equating. Research & Practice in Assessment, 8, 26–39. http://www.rpajournal.com/dev/wp-content/uploads/2013/05/SF3.pdf&sa=X&scisig=AAGBfm2n7WN_mLyfzwk1qYsjHvIek10RhA&oi=scholarr&ei=JU2iUtXYMJGrhQfYz4HQCA&ved=0CDAQgAMoATAA
• Morris, G. A., Harshman, N., Branum-Martin, L., Mazur, E., Mzoughi, T., & Baker, S. D. (2012). An item response curves analysis of the Force Concept Inventory. American Journal of Physics, 80(9), 825–831. https://doi.org/10.1119/1.4731618
• Neumann, I., Neumann, K., & Nehm, R. (2010). Evaluating instrument quality in science education: Rasch-based analyses of a Nature of Science Test. Taylor & Francis. https://doi.org/10.1080/09500693.2010.511297ï
• Nordin, H., & Ariffin, T. F. T. (2016). Validation of a technological pedagogical content knowledge instrument in a Malaysian secondary school context. Malaysian Journal of Learning and Instruction, 13(1), 1–24. https://doi.org/10.32890/mjli2016.13.1.1
• OECD. (2015). OECD iLibrary | PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy. https://www.oecd-ilibrary.org/education/pisa-2015-assessment-and-analytical-framework_9789264255425-en
• Planinic, M., Ivanjek, L., & Susac, A. (2010). Rasch model based analysis of the Force Concept Inventory. Physical Review Special Topics - Physics Education Research, 6(1). https://doi.org/10.1103/PhysRevSTPER.6.010103
• Pratiwi, M., Siahaan, P., Samsudin, A., Aminudin, A. H., & Rachmadtullah, R. (2020). Introduction , Connection , Application , Reflection , Extension-Multimedia Based Integrated Instruction ( ICARE-U ): A Model to Improve Creative Thinking Skills. 24(08).
• Ravand, H., & Firoozi, T. (2016). Examining construct validity of the master’s UEE using the Rasch model and the six aspects of the Messick's framework. International Journal of Language Testing, 6(1), 1-18.
• Romine, W. L., Schaffer, D. L., & Barrow, L. (2015). International Journal of Science Education Development and Application of a Novel Rasch-based Methodology for Evaluating Multi-Tiered Assessment Instruments: Validation and utilization of an undergraduate diagnostic test of the water cycle. Taylor & Francis, 37(16), 2740–2768. https://doi.org/10.1080/09500693.2015.1105398
• Rudolph, J. L., & Horibe, S. (2016). What do we mean by science education for civic engagement? Journal of Research in Science Teaching, 53(6), 805–820. https://doi.org/10.1002/tea.21303
• Runnels, J. (2012). Using the Rasch model to validate a multiple choice English achievement test. International Journal of Language Studies, 6(4), 141-155.
• Rusilowati, A., Kurniawati, L., Nugroho, S. E., & Widiyatmoko, A. (2016). Developing an instrument of scientific literacy asessment on the cycle theme. International Journal of Environmental and Science Education, 11(12), 5718–5727.
• Rusilowati, A., Nugroho, S. E., Susilowati, E. S. M., Mustika, T., Harfiyani, N., & Prabowo, H. T. (2018). The development of scientific literacy assessment to measure student’s scientific literacy skills in energy theme. Journal of Physics: Conference Series, 983(1). https://doi.org/10.1088/1742-6596/983/1/012046
• Saddhono, K., & Rohmadi, M. (2014). A sociolinguistics study on the use of the Javanese language in the learning process in primary schools in Surakarta, Central Java, Indonesia. International Education Studies, 7(6), 25–30. https://doi.org/10.5539/ies.v7n6p25
• Samsudin, A. (2020). Rasch Analysis: Measuring Students Attitudes toward Physics using the CLASS. Test Engineering & Management, 83(June), 15461–15467.
• Sheu, T. W., Tsai, C. P., Tzeng, J. W., Chen, T. L., & Nagai, M. (2013). An algorithm of the misconception order. Applied Mechanics and Materials, 284-287, 3010–3014. https://doi.org/10.4028/www.scientific.net/AMM.284-287.3010
• Shu, Z., Henson, R., & Luecht, R. (2013). Using Deterministic, Gated Item Response Theory Model to Detect Test Cheating due to Item Compromise. Psychometrika, 78(3), 481–497. https://doi.org/10.1007/s11336-012-9311-3
• Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26(1), 100-107.
• Smith, A. B., Fallowfield, L. J., Stark, D. P., Velikova, G., & Jenkins, V. (2010). A Rasch and confirmatory factor analysis of the General Health Questionnaire (GHQ) - 12. Health and Quality of Life Outcomes, 8. https://doi.org/10.1186/1477-7525-8-45
• Soobard, R., & Rannikmäe, M. (2011). Assessing student’s level of scientific literacy using interdisciplinary scenarios. In Science Education International, 22(2), 133-144. https://eric.ed.gov/?id=EJ941672
• Stenbeck, M., Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1992). Fundamentals of Item Response Theory. Contemporary Sociology, 21(2), 289. https://doi.org/10.2307/2075521
• Suhandi, A., & Samsudin, A. (2019). Effectiveness of the use of developed teacher’s book in guiding the implementation of physics teaching that provides science literacy and instill spiritual attitudes. Journal of Physics: Conference Series, 1280(5). https://doi.org/10.1088/1742-6596/1280/5/052054
• Sumintono, B. (2018). Rasch Model Measurements as Tools in Assesment for Learning. https://doi.org/10.2991/icei-17.2018.11
• Suryana, T. G. S., Setyadin, A. H., Samsudin, A., & Kaniawati, I. (2020). Assessing Multidimensional Energy Literacy of High School Students: An Analysis of Rasch Model. Journal of Physics: Conference Series, 1467(1), 1-10. https://doi.org/10.1088/1742-6596/1467/1/012034
• Susongko, P. (2016). Validation of science achievement test with the Rasch model. Jurnal Pendidikan IPA Indonesia, 5(2), 268–277. https://doi.org/10.15294/jpii.v5i2.7690
• Tamassia, L., & Frans, R. (2014). Does integrated science education improve scientific literacy? In Journal of the European Teacher Education Network, 9(1),131–141. http://62.28.241.73/index.php/jeten/article/view/44
• Wagner-Menghin, M., Preusche, I., & Schmidts, M. (2013). The Effects of Reusing Written Test Items: A Study Using the Rasch Model. ISRN Education, 2013, 1–7. https://doi.org/10.1155/2013/585420
• Wilson, K., Low, D., Verdon, M., & Verdon, A. (2016). Differences in gender performance on competitive physics selection tests. Physical Review Physics Education Research, 12(2), 020111. https://doi.org/10.1103/PhysRevPhysEducRes.12.020111
• Wind, S. A., & Gale, J. D. (2015). Diagnostic Opportunities Using Rasch Measurement in the Context of a Misconceptions-Based Physical Science Assessment. Science Education, 99(4), 721–741. https://doi.org/10.1002/sce.21172
• Winne, P. H. (2020). Construct and consequential validity for learning analytics based on trace data. Computers in Human Behavior, 112, 106457.
• Wright, B. D., & Stone, M. H. (1979). Best Test Design: Rasch Measurement. Chicago, Illinois: MESA Press
• Wright, B., & Mok, M. M. C. (2004). An Overview of the Family of Rasch Measurement Models. Introduction to Rasch Measurement, 1–24. http://www.statistica.unimib.it/utenti/lovaglio/overview rasch.pdf
• Wu, M., & Adams, R. (2007). Applyıng the Rasch model to psycho-socıal measurement a Practıcal Approach https://pdfs.semanticscholar.org/a263/e16ffe74da6ec87d7855ed878d6ab90acfed.pdf
• Yenni, R., Hernani, & Widodo, A. (2017). The implementation of integrated science teaching materials based socio-scientific issues to improve students scientific literacy for environmental pollution theme. AIP Conference Proceedings, 1848, 060002 https://doi.org/10.1063/1.4983970