Harmonizing perspectives to understand attitudes: A mixed methods approach to crafting an assessment literacy attitude scale

Abstract

Assessment literacy's vital role in faculty effectiveness within higher education lacks sufficient tools for measuring faculty attitudes on this matter. Employing a sequential mixed-methods approach, this study utilized the theory of planned behavior to develop the Assessment Literacy Attitude Scale (ALAS) and evaluate its psychometric properties within the U.S. higher education context. The qualitative phase involved a literature review of relevant studies and existing self-report measures, interviews with stakeholders, and panel reviews to shape initial item development. Following the establishment of a conceptual foundation and a comprehensive overview of the scale's construction, our study advanced to the quantitative stage that involves factor analytical and item response theory approaches using data from 260 faculty across three public universities in the U.S. Exploratory factor analysis (EFA) was employed initially to obtain preliminary insights into the scale's factorial structure and dimensionality. Confirmatory factor analysis (CFA) was subsequently applied with separate data and the findings largely supported the conclusions from the EFA. Exploratory and confirmatory factor analyses resulted in 15 items loading across two factors in a good model fit range. Finally, we used nonparametric item response theory (IRT) techniques based on Mokken Scale Analysis (MSA) to evaluate individual items for evidence of effective psychometric properties to support the interpretation of ALAS scores, including monotonicity, scalability, and invariant item ordering. The newly-developed scale shows promise in assessing faculty attitudes toward enhancing their assessment literacy.

Keywords

• Assessment literacy
• Higher education
• Scale development
• Mokken scale analysis
• Mixed-methods

References

1. Adam, S. (2004). Using learning outcomes: A consideration of the nature, role, application and implications for European education of employing “learning outcomes” at the local, national and international levels. Paper presented at the Bologna Seminar, Heriot-Watt University, Edinburgh United Kingdom. http://www.aic.lv/ace/ace_disk/Bologna/Bol_semin/Edinburgh/S_Adam_Bacgrerep_presentation.pdf Accessed on 16 November 2023.
2. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179-211. https://doi.org/10.1016/0749-5978(91)90020-T
3. Ajzen, I. (2001). Nature and operation of attitudes. Annual Review of Psychology, 52(1), 27-58. https://doi.org/10.1146/annurev.psych.52.1.27
4. Ajzen, I., & Timko, C. (1986). Correspondence between health attitudes and behavior. Basic and Applied Social Psychology, 7(4), 259 276. https://doi.org/10.1207/s15324834basp0704_2
5. Archie, T., Hayward, C.N., Yoshinobu, S., & Laursen, S.L. (2022). Investigating the linkage between professional development and mathematics instructors’ use of teaching practices using the theory of planned behavior. Plos One, 17(4), e0267097. https://doi.org/10.1371/journal.pone.0267097
6. Balloo, K., Norman, M., & Winstone, N.E. (2018, January). Evaluation of a large-scale inclusive assessment intervention: a novel approach to quantifying perceptions about assessment literacy. In The Changing Shape of Higher Education-Can Excellence and Inclusion Cohabit?: Conference Programmme and Book of Abstracts. University of Southern Queensland. https://srhe.ac.uk/arc/conference2018/downloads/SRHE_Conf_2018_Programme_Papers.pdf
7. Biggs, J., & Tang, C. (2011). Train-the-trainers: Implementing outcomes-based teaching and learning in Malaysian higher education. Malaysian Journal of Learning and Instruction, 8, 1-19.
8. Caspersen, J., & Smeby, J.C. (2018). The relationship among learning outcome measures used in higher education. Quality in Higher Education, 24(2), 117 135. https://doi.org/10.1080/13538322.2018.1484411

9. Chang, L. (1995). Connotatively consistent and reversed connotatively inconsistent items are not fully equivalent: Generalizability study. Educational and Psychological Measurement, 55(6), 991-997. https://doi.org/10.1177/0013164495055006007
10. Coates, H. (2016). Assessing student learning outcomes internationally: Insights and frontiers. Assessment & Evaluation in Higher Education, 41(5), 662 676. https://doi.org/10.1080/02602938.2016.1160273
11. Cochran, W.G. (1977). Sampling techniques. John Wiley & Sons.
12. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37 46. https://doi.org/10.1177/001316446002000104
13. Cole, K.L., Turner, R.C., & Gitchel, W.D. (2019). A study of polytomous IRT methods and item wording directionality effects on perceived stress items. Personality and Individual Differences, 147(6), 63-72. https://doi.org/10.1016/j.paid.2019.03.046
14. Conner, M., & Armitage, C.J. (1998). Extending the theory of planned behavior: A review and avenues for further research. Journal of Applied Social Psychology, 28(15), 1429-1464. https://doi.org/10.1111/j.1559-1816.1998.tb01685.x
15. Creswell, J.W., & Clark, V.P. (2011). Mixed methods research. SAGE Publications.
16. Crick, R.D., Broadfoot, P., & Claxton, G. (2004). Developing an effective lifelong learning inventory: The ELLI project. Assessment in Education: Principles, Policy & Practice, 11(3), 247-272. https://doi.org/10.1080/0969594042000304582
17. Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
18. Dann, R. (2014). Assessment as learning: blurring the boundaries of assessment and learning for theory, policy and practice. Assessment in Education: Principles, Policy & Practice, 21(2), 149-166. https://doi.org/10.1080/0969594X.2014.898128
19. Dilek, H., & Akbaş, U. (2022). Investigation of education value perception scale's psychometric properties according to CTT and IRT. International Journal of Assessment Tools in Education, 9(3), 548-564. https://doi.org/10.21449/ijate.986530
20. Dill, D. (2007). Quality assurance in higher education: Practices and issues. The 3rd International Encyclopedia of Education.
21. Dunn, R., Hattie, J., & Bowles, T. (2018). Using the Theory of Planned Behavior to explore teachers’ intentions to engage in ongoing teacher professional learning. Studies in Educational Evaluation, 59, 288-294. https://doi.org/10.1016/j.stueduc.2018.10.001
22. Eubanks, D. (2019). Reassessing the elephant, part 1. Assessment Update, 31(2), 6-7. https://doi.org/10.1002/au.30166
23. Evans, C. (2016). Enhancing assessment feedback practice in higher education: The EAT framework. University of Southampton. https://www.southampton.ac.uk/assets/imported/transforms/content block/UsefulDownloads_Download/A0999D3AF2AF4C5AA24B5BEA08C61D8E/EAT%20Guide%20April%20FINAL1%20ALL.pdf
24. Field, A. (2003). Discovering Statistics using IBM SPSS statistics. Sage Publications.
25. Fokkema, M., & Greiff, S. (2017). How performing PCA and CFA on the same data equals trouble: Overfitting in the assessment of internal structure and some editorial thoughts on it [Editorial]. European Journal of Psychological Assessment, 33(6), 399–402. https://doi.org/10.1027/1015-5759/a000460
26. Fornell, C., & David, F.L. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39 50. https://doi.org/10.2307/3151312
27. Henseler, J., Ringle, C.M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy Marketing Science, 43, 115–135. https://doi.org/10.1007/s11747-014-0403-8
28. Hines, S.R. (2009). Investigating faculty development program assessment practices: What's being done and how can it be improved?. The Journal of Faculty Development, 23(3), 5.
29. Holmboe, E.S., Ward, D.S., Reznick, R.K., Katsufrakis, P.J., Leslie, K.M., Patel, V.L., ... & Nelson, E.A. (2011). Faculty development in assessment: the missing link in competency-based medical education. Academic Medicine, 86(4), 460 467. https://doi.org/10.1097/acm.0b013e31820cb2a7
30. Hora, M.T., & Anderson, C. (2012). Perceived norms for interactive teaching and their relationship to instructional decision-making: A mixed methods study. Higher Education, 64, 573-592. https://doi.org/10.1007/s10734-012-9513-8
31. Howard, M.C. (2016). A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve?. International Journal of Human Computer Interaction, 32(1), 51 62. https://doi.org/10.1080/10447318.2015.1087664
32. Hu, L.T., & Bentler, P.M. (1999). Cutof criteria for fit indexes in covariance structural analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
33. Jankowski, N.A., & Marshall, D.W. (2017). Degrees that matter: Moving higher education to a learning systems paradigm. Routledge. https://doi.org/10.4324/9781003444015
34. Kao, C.P., Lin, K.Y., & Chien, H.M. (2018). Predicting teachers’ behavioral intentions regarding web-based professional development by the theory of planned behavior. EURASIA Journal of Mathematics, Science and Technology Education, 14(5), 1887-1897. https://doi.org/10.29333/ejmste/85425
35. Kline, P. (1994). An easy guide to factor analysis. Routledge.
36. Knauder, H., & Koschmieder, C. (2019). Individualized student support in primary school teaching: A review of influencing factors using the Theory of Planned Behavior (TPB). Teaching and Teacher Education, 77, 66-76. https://doi.org/10.1016/j.tate.2018.09.012
37. Kremmel, B., & Harding, L. (2020). Towards a comprehensive, empirical model of language assessment literacy across stakeholder groups: Developing the language assessment literacy survey. Language Assessment Quarterly, 17(1), 100 120. https://doi.org/10.1080/15434303.2019.1674855
38. Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310
39. Ligtvoet, R., Van der Ark, L.A., Marvelde, J.M. te, & Sijtsma, K. (2010). Investigating an Invariant Item Ordering for Polytomously Scored Items. Educational and Psychological Measurement, 70(4), 578–595. https://doi.org/10.1177/0013164409355697
40. Liu, O.L., Bridgeman, B., & Adler, R.M. (2012). Measuring learning outcomes in higher education: Motivation matters. Educational Researcher, 41(9), 352 362. https://doi.org/10.3102/0013189X12459679
41. Madigan, D.J., & Kim, L.E. (2021). Towards an understanding of teacher attrition: A meta-analysis of burnout, job satisfaction, and teachers’ intentions to quit. Teaching and Teacher Education, 105, 103425. https://doi.org/10.1016/j.tate.2021.103425
42. Mazza, A., Punzo, A., & McGuire, B. (2014). KernSmoothIRT: An R package for kernel smoothing in Item Response Theory. Journal of Statistical Software, 58(6). https://doi.org/10.18637/jss.v058.i06
43. McDonald, R.P. (1999). Test theory: A unified treatment. Taylor & Francis.
44. McKown, C., Kharitonova, M., Russo-Ponsaran, N.M., & Aksu-Dunya, B. (2023). Development and Validation of a shortened form of SELweb EE, a Web-Based Assessment of Children’s Social and Emotional Competence. Assessment, 30(1), 171-189. https://doi.org/10.1177/10731911211046044
45. Medland, E. (2019). ‘I’m an assessment illiterate’: Towards a shared discourse of assessment literacy for external examiners. Assessment & Evaluation in Higher Education, 44(4), 565-580. https://doi.org/10.1080/02602938.2018.1523363
46. Meijer, R.R., & Baneke, J.J. (2004). Analyzing psychopathology items: A case for Nonparametric Item Response Theory Modeling. Psychological Methods, 9(3), 354–368. https://doi.org/10.1037/1082-989X.9.3.354
47. Meijer, R.R., Tendeiro, J.N., & Wanders, R.B.K. (2015). The use of nonparametric item response theory to explore data quality. In S.P. Reise & D.A. Revicki (Eds.), Handbook of Item Response Theory Modeling: Applications to typical performance assessment (pp. 85–110). Routledge.
48. Mokken, R.J. (1971). A theory and procedure of scale analysis. De Gruyter.
49. Morell, L., & Tan, R.J.B. (2009). Validating for use and interpretation: A mixed methods contribution illustrated. Journal of Mixed Methods Research, 3(3), 242 264. https://doi.org/10.1177/1558689809335079
50. Muthén, B.O. (1993). Goodness of fit with categorical and other nonnormal variables. In K.A. Bollen, & J.S. Long (Eds.), Testing structural equation models (pp. 205-234). Sage Publishing.
51. O’Neill, G., McEvoy, E., & Maguire, T. (2023). Supporting assessment literacy in changing times. In C. Evans and M. Waring (Eds.), Research handbook on innovations in assessment and feedback in higher education. Elgar Publishing.
52. Padilla, J.L., & Leighton, J.P. (2017). Cognitive interviewing and think aloud methods. In B. Zumbo & A. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 211-228). Springer.
53. Pastore, S. (2022). Assessment Literacy in the higher education context: A critical review. Intersection: A Journal at the Intersection of Assessment and Learning, 4(1). https://doi.org/10.61669/001c.39702
54. Pastore, S., & Andrade, H.L. (2019). Teacher assessment literacy: A three-dimensional model. Teaching and Teacher Education, 84, 128 138. https://doi.org/10.1016/j.tate.2019.05.003
55. Pett, M.A., Lackey, N.R., & Sullivan, J.J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. Sage Publications.
56. Price, M., Rust, C., ODonovan, B., Handley, K., & Bryant, R. (2012). Assessment literacy: The foundation for improving student learning. ASKe, Oxford Centre for Staff and Learning Development.
57. R Core Team (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
58. Ramsay, J.O., & Silverman, B.W. (2005). Functional data analysis (2nd ed.). Springer.
59. Revelle, W. (2023). Psych: Procedures for psychological, psychometric, and personality research. Northwestern University, Evanston, Illinois. R package version 2.3.9, https://CRAN.R-project.org/package=psych
60. Rosseel, Y. (2012). lavaan: An R package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
61. Rimal, R.N., & Real, K. (2003). Understanding the influence of perceived norms on behaviors. Communication Theory, 13(2), 184 203. https://doi.org/10.1111/j.1468 2885.2003.tb00288.x
62. Sadler, D.R. (2017). Academic achievement standards and quality assurance. Quality in Higher Education, 23(2), 81-99. https://doi.org/10.1080/13538322.2017.1356614
63. Scholl, K., & Olsen, H.M. (2014). Measuring student learning outcomes using the SALG instrument. SCHOLE: A Journal of Leisure Studies and Recreation Education, 29(1), 37-50. https://doi.org/10.1080/1937156X.2014.11949710
64. Sijtsma, K., & Molenaar, I.W. (2002). Introduction to nonparametric item response theory (Vol. 5). Sage Publications.
65. Sijtsma, K., & van der Ark, L.A. (2017). A tutorial on how to do a Mokken scale analysis on your test and questionnaire data. British Journal of Mathematical and Statistical Psychology, 70(1), 137–158. https://doi.org/10.1111/bmsp.12078
66. Singh, M., & Ramya, K.R. (2011). Outcome based education. International Journal of Nursing Education, 3(2), 87-91.
67. Sliter, K.A., & Zickar, M.J. (2014). An IRT examination of the psychometric functioning of negatively worded personality items. Educational and Psychological Measurement, 74(2), 214-226. https://doi.org/10.1177/0013164413504584
68. Tabachnick, B., & Fidell, L.S. (1996). Using multivariate statistics. Harper Collins.
69. Van der Ark, L.A. (2007). Mokken scale analysis in R. Journal of Statistical Software, 20, 1-19. https://doi.org/10.18637/jss.v020.i11
70. Van der Ark, L.A. (2012). New developments in Mokken scale analysis in R. Journal of Statistical Software, 48, 1-27. https://doi.org/10.18637/jss.v048.i05
71. Velicer, W.F., Eaton, C.A., & Fava, J.L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R.D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment (pp. 41-71). Kluwer.
72. Williams, J. (2016). Quality assurance and quality enhancement: Is there a relationship?. Quality in Higher Education, 22(2), 97 102. https://doi.org/10.1080/13538322.2016.1227207
73. Wolf, R., Zahner, D., & Benjamin, R. (2015). Methodological challenges in international comparative post-secondary assessment programs: Lessons learned and the road ahead. Studies in Higher Education, 40(3), 471 481. https://doi.org/10.1080/03075079.2015.1004239
74. Wright, B.D., & Masters, G.N. (1982). Rating Scale Analysis. MESA Press.
75. Wright, S.L., & Jenkins-Guarnieri, M.A. (2023). Further validation of the social efficacy and social outcome expectations scale. Journal of Psychoeducational Assessment, 42(1), 74-88. https://doi.org/10.1177/07342829231198277
76. Xu, Y., & Brown, G.T. (2016). Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education, 58, 149 162. https://doi.org/10.1016/j.tate.2016.05.010
77. Zhou, Y. (2019). A mixed methods model of scale development and validation analysis. Measurement: Interdisciplinary Research and Perspectives, 17(1), 38 47. https://doi.org/10.1080/15366367.2018.1479088
78. Zhu, X., & Evans, C. (2022). Enhancing the development and understanding of assessment literacy in higher education. European Journal of Higher Education, 1 21. https://doi.org/10.1080/21568235.2022.2118149
79. Zoom Video Communications, Inc. (2023). ZOOM cloud meetings (Version 5.15.5). https://zoom.com