Exploring the psychometric properties of the Testwiseness Rating Scale using a measurement modeling approach

Abstract

Testwiseness refers to the ability of a test-taker or examinee to use strategic knowledge and cues within a test in order to improve their performance. These skills, cues, and knowledge are often independent of the content being assessed. Testwiseness includes skills such as using the process of elimination, recognizing patterns in answer choices, paying attention to grammatical errors, and managing time effectively. Testwiseness is important because it influences test outcomes, and potentially gives advantages to students with high testwiseness skills, rather than solely reflecting their true ability or knowledge level. Thus, understanding testwiseness is crucial for ensuring meaningful interpretation and use of test scores. The current study describes the development and psychometric evaluation of the Testwiseness Rating Scale (TWRS). We collected pilot data for the TWRS from a sample of 300 students. We evaluated the psychometric properties using a combination of factor analysis and Rasch measurement theory techniques, including differential item functioning analysis related to student gender. Overall, we found evidence to support the interpretation and use of scores from the TWRS to measure students’ testwiseness. We consider the implications of our findings for research and practice and discuss directions for future research related to this instrument.

Keywords

• Testwiseness
• Psychometric properties
• DIF
• Rasch model
• Partial credit model

References

1. Abedalqader, O., Hisham, B.K., Abedalqader, B., & Hassan, B.H (2015). The level of testwiseness for the students of Arts and Science Faculty at Sharourah and its relationship with some variables. Journal Education and Practice, 6(29), 102-254. https://files.eric.ed.gov/fulltext/EJ1081246.pdf
2. Aborisade, O.J., & Adeniji, P.O., (2022). Effects of testwiseness strategy on the academic performance of senior secondary school students in economic in Oyo State, Nigeria. Open Journal of Educational Development, 3(2), 01-12. https://doi.org/10.52417/ojed.v3i2.368
3. Adams, R.J., Wu, M.L., & Wilson, M. (2012). The Rasch rating model and the disordered threshold controversy. Educational and Psychological Measurement, 72, 547 573. https://doi.org/10.1177/001316441143
4. Allan, A. (1992). Development and validation of a scale to measure test-wiseness in EFL/ESL reading test takers. Language Testing, 9(2), 101-119. https://doi.org/10.1177/026553229200900201
5. Alnasraween, M.S., Alsmadi, M.S., Al-zboon, H.S., & Alkurshe, T.O. (2022). The level of universities students’ test wiseness in Jordan during distance learning in light of some variables. Hindawi Educational Research International, 1-10. https://doi.org/10.1155/2022/6381857
6. Altindis, N., Ocheni, C.A., Tong, Y., & Obafemi, K. (2025). Impacts of gender, engineering, and role models on high school students’ overall STEM interest and perceptions of engineering. Education Sciences, 15(9), Article 1217. https://doi.org/10.3390/educsci15091217
7. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
8. Bond, T.G., & Fox, C.M. (2015). Applying the Rasch Model: Fundamental measurement in the human sciences (3rd ed.). Routledge.

9. Brown T.A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford Press
10. Browne, M.W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230-258. https://doi.org/10.1177/0049124192021002005
11. Bumbalkova, E. (2021). Test-taking strategies in second language receptive skills tests. a literature review. International Journal of Instruction, 14(2), 647 664. https://doig.org/10.29333/iji.2021.14236a
12. Chen, W.H., Revicki, D. (2014). Differential item functioning (DIF). In A.C. Michalos (Ed.) Encyclopedia of quality of life and well being research. Springer. https://doi.org/10.1007/978-94-007-0753-5_728
13. DeVore, S., Stewart, J., & Stewart, G. (2016). Examining the effects of testwiseness in conceptual physics evaluations. Physical Review Physics Education Research, 12, Article 020138. https://doi.org/10.1103/PhysRevPhysEducRes.12.020138
14. Dodeen, H. (2008). Assessing test-taking strategies of university students: Developing a scale and estimating its psychometric indices. Assessment & Evaluation in Higher Education, 33(4), 409-419. https://doi.org/10.1080/02602930701562874
15. Ebel, R.L., & Frisbie, D.A. (1991). Essentials of educational measurement (6th ed.). Prentice Hall.
16. Engelhard, G. (2013). Invariant measurement: Using Rasch models in the social, behavioral and health sciences. Routledge.
17. Flavell, J.H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906-911. https://doi.org/10.1037/0003-066X.34.10.906
18. Gbore, L.O., & Osakuade, J.O. (2016). Effects of testwiseness training in mathematics on adolescent secondary school students’ test anxiety in Ondo State, Nigeria. Journal of Education and Practice, 7(10). 34-39. https://files.eric.ed.gov/fulltext/EJ1099554.pdf
19. Hammadneh, I. (2011). The degree to which students at the university level use test-wise strategies. Manara Magazine, Al al-Bayt University, 17(1), 291-312.
20. Hayati, A.M., & Ghojogh, A.N. (2008). Investigating the influence of proficiency and gender on the use of selected test-wiseness strategies in higher education. English Language Teaching, 1(1), 169-181 https://pdfs.semanticscholar.org/7253/29381830008dca77f71617c36816bb3fce1c.pdf
21. Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
22. Kelton, M.L., LeardMann, C.A., Smith, B., Boyko, E.J., Hooper, T.I., Gackstetter, G.D., Bliese, P.D., Hoge, C.W., Smith, T.C., & Millennium Cohort Study Team (2010). Exploratory factor analysis of self-reported symptoms in a large, population-based military cohort. BMC Medical Research Methodology, 10, Article 94. https://doi.org/10.1186/1471-2288-10-94
23. Kline, R.B. (2023). Principles and practice of structural equation modeling (5th ed.). Guilford Press.
24. Linacre, J.M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85-106.
25. Mair, P., & Hatzinger, R. (2007). Extended Rasch Modeling: The eRm package for the application of IRT models in R. Journal of Statistical Software, 20(9), 1-20. https://doi.org/10.18637/jss.v020.i09
26. Marsh, H.W., Ludtke, O., Muthen, B., Asparouhov, T., Morin, A.J., Trautwein, U., & Nagengast, B. (2009). A new look at the Big five factor structure through explanatory structural equation modeling. Psychological Assessment, 22(3), 471-491. https://doi.org/10.1037/a0019227
27. Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). American Council on Education and Macmillan.
28. Millman, J., Bishop, C.H., & Ebel, R. (1965). An analysis of test-wiseness. Educational and Psychological Measurement, 25(3), 707-726. https://doi.org/10.1177/001316446502500304
29. Millsap, R.E. (2011). Statistical approaches to measurement invariance, Routledge.
30. Ocheni, C.A., & Mujahid, T. (2023). Analysis of the predictive power of testwiseness and learners’ autonomy on students’ achievement in projectile motion [Unpublished manuscript]. University of Alabama, Tuscaloosa.
31. Ocheni, C.A., Okeke, A.O., Agah, J.J., Oguguo, B.C.E., Ene, C.U., Ezuguw, I.J., & Asongo, S.T. (2024). Moderation of gender and school location on the prediction of test anxiety on physics students’ academic achievement in waves motion. Physics Education Research, 6(1), 11 20. https://doi.org/10.21580/perj.2024.6.1.20668
32. Okeke, A.O., Ocheni, C.A., Oguguo, B.C.E., & Asongo, S.T. (2022). Test anxiety and academic stress as predictors of secondary school students’ academic achievement in Physics. International Journal of Social Sciences & Educational Studies, 9(4), 172-182. https://doi.org/10.23918/ijsses.v9i4p172
33. Padilla, J.L., & Leighton, J.P. (2017). Cognitive interviewing and think aloud methods. In B.D. Zumbo & A.M. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 211–228). Springer International Publishing. https://doi.org/10.1007/978-3-319-56129-5_12
34. Pallant, J. (2013). Checking the reliability of a scale: SPSS survival manual (5th ed.). Routledge
35. Peabody, M.R., & Wind, S.A. (2019). Exploring the stability of different item functioning across administrations and critical values using the Rasch separate calibration t-test method. Measurement: Interdisciplinary Research and Perspectives, 17(2), 78 92. https://doi.org/10.1080/15366367.2018.1533782
36. Rammstedt, B., & John, O.P. (2007). Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41(1), 203-212. https://doi.org/10.1016/j.jrp.2006.02.001
37. Robitzsch, A., Kiefer, T., & Wu, M. (2024). TAM: Test Analysis Module. R package version 4.2-21. https://CRAN.R-project.org/package=TAM
38. Russo, J.A. (2019). The impact of a short test-wiseness intervention on standardized numeracy assessment scores: A cautionary tale about using NAPLAN growth data to evaluate primary schools, networks: An Online Journal for Teacher Research, 21(2). https://doi.org/10.4148/2470-6353.1301
39. Smith, G.T., McCarthy, D.M., & Anderson, K.G. (2000). On the sins of short-form development. Psychological Assessment, 12(1), 102-111. https://doi.org/10.1037/1040-3590.12.1.102
40. Stankov, L., & Lee, J. (2014). Quest for the best non-cognitive predictor of academic achievement. Educational Psychology, 34(1), 1-8. https://doi.org/10.1080/01443410.2013.858908
41. Thomas, G.B., & Koller, O. (2018). Testwiseness: A neglected construct? Empirical findings on testing and trainability of testwiseness. Journals in Science, Humanities, Technical and Medical Publishing, 8(1), 63-80. https://doi.org/10.1007/s35834-018-0204-0
42. Wind, S.A., Mansouri, B., & Jami, P.Y. (2019). Student perceptions of grammar instruction in Iranian secondary education: Evaluation of an instrument using Rasch measurement theory. Journal of Applied Measurement, 20(1), 46-65.
43. Wright, B.D., & Masters, G.N. (1982). Rating scale analysis: Rasch measurement. MESA Press.
44. Zwick, R. (2002). Fair game? The use of standardized admissions tests in higher education. Routledge Falmer.