Validity Evidence for the Perceptions of Secondary School Students of ‘What Research is’ Scale and Measurement Invariance

Abstract

Research is a concrete action in academia which has uplifted societies’ prosperity. Although researchers have given particular attention to student perceptions about what research is in a higher education context, little attention has been given to secondary school students’ perceptions about this issue. To fill this gap, Yeoman et al. (2016) qualitatively developed an instrument measuring secondary school students’ perceptions of what research is. The present study quantitatively validates this scale using the dataset originally used to qualitatively validate it. The factor structure of the ‘what research is’ scale and measurement invariance across gender, school type, and key stage was examined. The sample is composed of 2634 secondary school students in seven schools located in East Anglia, UK. The data from this original sample showed a relatively acceptable fit to the four-factor structure after omitting some items. The result also highlighted that whilst there was evidence on configural and metric level invariance (i.e. the factor structures and the factor loadings of the scale are equivalent across gender, school type, and key stage), scalar level invariance was not met (i.e. the item intercepts of the scale are not equivalent across gender, school type, and key stage). Recommendations for future studies and future directions for research are discussed.

Keywords

• Research
• what research is
• secondary school
• student perception
• scale validation
• measurement invariance

Validity Evidence for the Perceptions of Secondary School Students of ‘What Research is’ Scale and Measurement Invariance

Abstract

Research is a concrete action in academia which has uplifted societies’ prosperity. Although researchers have given particular attention to student perceptions about what research is in a higher education context, little attention has been given to secondary school students’ perceptions about this issue. To fill this gap, Yeoman et al. (2016) qualitatively developed an instrument measuring secondary school students’ perceptions of what research is. The present study quantitatively validates this scale using the dataset originally used to qualitatively validate it. The factor structure of the ‘what research is’ scale and measurement invariance across gender, school type, and key stage was examined. The sample is composed of 2634 secondary school students in seven schools located in East Anglia, UK. The data from this original sample showed a relatively acceptable fit to the four-factor structure after omitting some items. The result also highlighted that whilst there was evidence on configural and metric level invariance (i.e. the factor structures and the factor loadings of the scale are equivalent across gender, school type, and key stage), scalar level invariance was not met (i.e. the item intercepts of the scale are not equivalent across gender, school type, and key stage). Recommendations for future studies and future directions for research are discussed.

Keywords

• Research
• What research is
• Student perception
• Scale validation
• Measurement invariance

References

1. Åkerlind, G. S. (2008). An academic perspective on research and being a researcher: An integration of the literature. Studies in Higher Education, 33(1), 17 31. https://doi.org/10.1080/03075070701794775
2. Archer, L., Osborne, J., DeWitt, J., Dillon, J., Wong, B., & Willis, B. (2013). ASPIRES: Young People’s Science and Career Aspirations, age 10 14. https://www.kcl.ac.uk/ecs/research/aspires/aspires-final-report-december-2013.pdf
3. Archer, L., Moote, J., MacLeod, E., Francis, B., & DeWitt, J. (2020). ASPIRES 2: Young people’s science and career aspirations, age 10-19. UCL Institute of Education. https://discovery.ucl.ac.uk/id/eprint/10092041/15/Moote_9538%20UCL%20Aspires%202%20report%20full%20online%20version.pdf
4. Bandura, A. (2006). Adolescent development from an agentic perspective. In: Pajares F and Urdan T (eds) Self-Efficacy Beliefs of Adolescents. Information Age Publishing, pp. 1–43.
5. Bazley, S. (2019). Ensuring Societal Advancement through Science and Technology: Pathways to Scientific Integration. CUSPE Communications https://doi.org/10.17863/CAM.38893
6. Bills, D. (2004). Supervisors' conceptions of research and the implications for supervisor development. International Journal for Academic Development, 9(1), 85-97. https://doi.org/10.1080/1360144042000296099
7. Brew, A. (2001). Conceptions of research: A phenomenographic study. Studies in higher education, 26(3), 271-285. https://doi.org/10.1080/03075070120076255
8. Britner, S. L., & Pajares, F. (2006). Sources of science self-efficacy beliefs of middle school students. Journal of Research in Science Teaching, 43, 485 499. https://doi.org/10.1002/tea.20131

9. Butz, A. R., & Usher, E. L. (2015). Salient sources of self-efficacy in reading and mathematics. Contemporary Educational Psychology, 42, 49 61. https://doi.org/10.1016/j.cedpsych.2015.04.001
10. Buuren, S. V., & Groothuis-Oudshoorn, K. (2010). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1 68. https://doi.org/10.18637/jss.v045.i03
11. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464-504. https://doi.org/10.1080/10705510701301834
12. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modelling, 9(2), 233 255. https://doi.org/10.1207/S15328007SEM0902_5
13. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates.
14. Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
15. Çaparlar, C. Ö., & Dönmez, A. (2016). What is scientific research and how can it be done?. Turkish Journal of Anaesthesiology and Reanimation, 44(4), 212. https://doi.org/10.5152/TJAR.2016.34711
16. DeWitt, J., & Archer, L. (2015). Who aspires to a science career? A comparison of survey responses from primary and secondary school students. International Journal of Science Education, 37(13), 2170-2192. https://doi.org/10.1080/09500693.2015.1071899
17. Donghong, C., & Shunke, S. (2008). The more, the earlier, the better: Science communication supports science education. In Communicating science in social contexts (pp. 151-163). Springer, Dordrecht.
18. Epskamp, S. (2015). semPlot: Unified visualizations of structural equation models. Structural Equation Modeling: a Multidisciplinary Journal, 22(3), 474 483. https://doi.org/10.1080/10705511.2014.937847
19. Fennema, E., & Sherman, J.A. (1976). Fennema-Sherman Mathematics Attitudes Scales: Instruments designed to measure attitudes toward the learning of mathematics by females and males. Journal Research Mathematics Education, 7(5), 324 326. https://doi.org/10.2307/748467
20. Georghiou, L. (2015). Value of research. Policy Paper by the Research, Innovation, and Science Policy Experts (RISE), European Commission. https://ec.europa.eu/research/innovation-union/pdf/expert-groups/rise/georghiou-value_research.pdf
21. Grever, M., Haydn, T., & Ribbens, K. (2008). Identity and school history: The perspective of young people from the Netherlands and England. British Journal of Educational Studies, 56(1), 76-94. https://doi.org/10.1111/j.1467-8527.2008.00396.x
22. Griffioen, D. M. (2020). A questionnaire to compare lecturers’ and students’ higher education research integration experiences. Teaching in Higher Education, AHEAD-OF-PRINT, 1-16. https://doi.org/10.1080/13562517.2019.1706162
23. Griffioen, D. M. (2019). The influence of undergraduate students’ research attitudes on their intentions for research usage in their future professional practice. Innovations in Education and Teaching International, 56(2), 162 172. https://doi.org/10.1080/14703297.2018.1425152
24. Griffioen, D. M. (2020). Differences in students’ experiences of research involvement: study years and disciplines compared. Journal of Further and Higher Education, 44(4), 454-466. https://doi.org/10.1080/0309877X.2019.1579894
25. Griffioen, D. M., & de Jong, U. (2015). Implementing research in professional higher education: Factors that influence lecturers’ perceptions. Educational Management Administration & Leadership, 43(4), 626 645. https://doi.org/10.1177/1741143214523008
26. He, J., & van de Vijver, F. (2012). Bias and equivalence in cross-cultural research. Online Readings in Psychology and Culture, 2(2), 2307-0919. https://doi.org/10.9707/2307-0919.1111
27. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
28. Hughes, R. A., White, I. R., Seaman, S. R., Carpenter, J. R., Tilling, K., & Sterne, J. A. (2014). Joint modelling rationale for chained equations. BMC Medical Research Methodology, 14(1), 28. https://doi.org/10.1186/1471-2288-14-28
29. Jöreskog, K. G. (1971). Simultaneous factor analysis in several populations. Psychometrika, 36(4), 409-426. https://doi.org/10.1007/BF02291366
30. Kelly, U., McNicoll, I., & White, J. (2014). The impact of universities on the UK economy. http://www.universitiesuk.ac.uk/highereducation/Documents/2014/TheImpactOfUniversitiesOnThe UkEconomy.pdf
31. Kiley, M., & Mullins, G. (2005). Supervisors' conceptions of research: What are they?. Scandinavian Journal of Educational Research, 49(3), 245 262. https://doi.org/10.1080/00313830500109550
32. Kline, R.B., (2011). Principles and Practices of Structural Equation Modelling. 3rd ed. The Guilford Press.
33. Mejía-Rodríguez, A. M., Luyten, H., & Meelissen, M. R. (2020). Gender Differences in Mathematics Self-concept Across the World: an Exploration of Student and Parent Data of TIMSS 2015. International Journal of Science and Mathematics Education, Advance online publication,1-22. https://doi.org/10.1007/s10763-020-10100-x
34. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525-543. https://doi.org/10.1007/BF02294825
35. Meyer, J. H., Shanahan, M. P., & Laugksch, R. C. (2005). Students' Conceptions of Research. I: A qualitative and quantitative analysis. Scandinavian Journal of Educational Research, 49(3), 225-244. https://doi.org/10.1080/00313830500109535
36. Meyer, J. H., Shanahan, M. P., & Laugksch, R. C. (2007). Students' conceptions of research. 2: An exploration of contrasting patterns of variation. Scandinavian Journal of Educational Research, 51(4), 415-433. https://doi.org/10.1080/00313830701485627
37. Moore, N., & Hooley, T. (2012). Talking about career: the language used by and with young people to discuss life, learning and work. Derby: iCeGS, University of Derby. https://derby.openrepository.com/bitstream/handle/10545/220535/Final%20Talking%20about%20career%20iCeGS%20Occasional%20Paper%2015062012%20NPM.pdf?sequence=8&isAllowed=y
38. Mosher, D. A. (2018). The effect of mode of presentation, cognitive load, and individual differences on recall [Doctoral dissertation, University of Reading]. http://centaur.reading.ac.uk/84822/
39. Nishimura H., Kanoshima E., Kono K. (2019). Advancement in Science and Technology and Human Societies. In: Abe S., Ozawa M., Kawata Y. (eds) Science of Societal Safety. Trust (Interdisciplinary Perspectives), vol 2. Springer, Singapore. https://doi.org/10.1007/978-981-13-2775-9_2
40. OECD (2015). Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris. http://dx.doi.org/10.1787/9789264239012-en
41. Ommering, B. W., Wijnen-Meijer, M., Dolmans, D. H., Dekker, F. W., & van Blankenstein, F. M. (2020). Promoting positive perceptions of and motivation for research among undergraduate medical students to stimulate future research involvement: a grounded theory study. BMC Medical Education, 20(1), 1-12. https://doi.org/10.1186/s12909-020-02112-6
42. Pearson, R. C., Crandall, K. J., Dispennette, K., & Maples, J. M. (2017). Students’ Perceptions of an Applied Research Experience in an Undergraduate Exercise Science Course. International Journal of Exercise Science, 10(7), 926-941.
43. Pitcher, R. (2011). Doctoral students’ conceptions of research. The Qualitative Report, 16(4), 971-983. Retrieved from http://www.nova.edu/ssss/QR/QR16-4/pitcher.pdf.
44. Pitcher, R., & Åkerlind, G. S. (2009). Postdoctoral researchers’ conceptions of research: A metaphor analysis. The International Journal for Researcher Development, 1, 42-56.
45. R Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/
46. Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5 12 (BETA). Journal of Statistical Software, 48(2), 1 36. https://doi.org/10.1108/1759751X201100009
47. Saleem, M. A., Eagle, L., Akhtar, N., & Wasaya, A. (2020). What do prospective students look for in higher degrees by research? A scale development study. Journal of Marketing for Higher Education, 30(1), 45-65. https://doi.org/10.1080/08841241.2019.1678548
48. Salter, A. J., & Martin, B. R. (2001). The economic benefits of publicly funded basic research: a critical review. Research Policy, 30(3), 509-532. https://doi.org/10.1016/S0048-7333(00)00091-3
49. Santos, M. S., Martins, J. V., Silva, A. P. F., Paula, F. G., Domingos, Á., & dos Santos, W. J. (2017). Analysis of the Influence of Undergraduate Research on the Engineering Formation from the Point of View of Students. International Journal of Science and Engineering Investigations, 66(6), 45-51.
50. Schmidt, J. A., Kafkas, S. S., Maier, K. S., Shumow, L., & Kackar-Cam, H. Z. (2019). Why are we learning this? Using mixed methods to understand teachers’ relevance statements and how they shape middle school students’ perceptions of science utility. Contemporary Educational Psychology, 57, 9-31. https://doi.org/10.1016/j.cedpsych.2018.08.005
51. Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. Allyn & Bacon.
52. Toma, R. B., & Greca, I. M. (2018). The effect of integrative STEM instruction on elementary students’ attitudes toward science. Eurasia Journal of Mathematics, Science and Technology Education, 14(4), 1383-1395. https://doi.org/10.29333/ejmste/83676
53. Vanderberg, R. J., & Lance, C. E. (2000). A Review and Synthesis of the Measurement Invariance Literature: Suggestions Practices, and Recommendations for Organizational Research. Organizational Research Methods, 3(1), 4 70. https://doi.org/10.1177/109442810031002
54. Verburgh, A., & Elen, J. (2011). The role of experienced research integration into teaching upon students' appreciation of research aspects in the learning environment. International Journal of University Teaching and Faculty Development, 1(4), 1-14.
55. Visser-Wijnveen, G. J., van der Rijst, R. M., & van Driel, J. H. (2016). A questionnaire to capture students’ perceptions of research integration in their courses. Higher Education, 71(4), 473-488. https://doi.org/10.1007/s10734-015-9918-2
56. Webb-Williams, J. (2018). Science self-efficacy in the primary classroom: Using mixed methods to investigate sources of self-efficacy. Research in Science Education, 48(5), 939-961. https://doi.org/10.1007/s11165-016-9592-0
57. Wikoff, R.L., & Buchalter, B.D. (1986). Factor analysis of four Fennema-Sherman mathematics attitude scales. International Journal Mathematics Education Science Technology, 17(6), 703-706. https://doi.org/10.1080/0020739860170605
58. Yeoman, K., Bowater, L., & Nardi, E. (2016). The representation of scientific research in the national curriculum and secondary school pupils’ perceptions of research, its function, usefulness and value to their lives [version 2; peer review: 2 approved]. F1000Research, 4, 1442. https://doi.org/10.12688/f1000research.7449.2
59. Yeoman, K., Nardi, E., Bowater, L., & Nguyen, H. (2017). ‘Just Google It?’: Pupils’ Perceptions and Experience of Research in the Secondary Classroom. British Journal of Educational Studies, 65(3), 281-305. https://doi.org/10.1080/00071005.2017.1310179