An Application of the Response Vector for Mastery Method of Standard Setting in Comparison with the Extended-Angoff and Cluster Analysis Methods

Abstract

This paper presents a real-data application of a recently developed method of standard setting, referred to as the response vector for mastery (RVM) method, and compares its performance with the popular Angoff method (an extended version) and cluster analysis method (CAM). The RVM enables the derivation of cut-scores on the bounded D-scale (ranging from 0 to 1), under the D-scoring method of measurement, and the logit scale of item response theory (IRT). Using a 25-item multiple-choice test in the field of educational measurement, the study demonstrates that the RVM method offers key advantages over Angoff and CAM methods, including reduced judgmental burden for panelists, greater classification accuracy and consistency, and flexibility in computing cut-scores at the domain (content area) level. Given its novel structure and promising empirical performance, the RVM method holds strong potential for researchers and practitioners seeking practical, defensible, and scalable approaches to standard setting in educational assessment contexts.

Keywords

• D-scoring method of measurement
• item response theory
• standard setting
• cut-scores
• classification accuracy
• classification consistency

References

1. Alshawwa, L. (2023). Standard setting: A review of methods. Asian Journal of Education and Social Studies, 42(2), 1-7. https://doi.org/10.9734/ajess/2023/v42i2909
2. Anderberg, M. R. (1973). Cluster analysis for applications. Academic Press.
3. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508–600). American Council on Education.
4. Akour, M., & Al-Omari, H. (2013), Empirical investigation of the stability of IRT item-parameters estimation. International Online Journal of Educational Sciences, 5(2), 291-301.
6. Atanasov, D. V., & Dimitrov, D. M. (2018). DELTA: A computer program for test scoring, equating, and item analysis under the D-scoring method. Riyadh, Saudi Arabia: Education & Training Evaluation Commission.
8. Binici, S., & Cuhadar, I. (2022). Validating performance standards via latent class analysis. Journal of Educational Measurement, 59(4), 502-516. https://doi.org/10.1111/jedm.12325

9. Bollen, K. A. (1989). Structural equations with latent variables. Wiley.
10. Brandon, P. R. (2004). Conclusions about frequently studied modified Angoff standard-setting topics. Applied Measurement in Education, 17(1), 59-88. https://doi.org/10.1207/s15324818ame1701_4
11. Brennan, R.L., & Lockwood, R.E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4 (2), 219-240. https://doi.org/10.1177/014662168000400209
12. Brown, R. S. (2007). Using latent class analysis to set academic performance standards. Educational Assessment, 12(3-4), 283–301. https://doi.org/10.1080/10627190701578321
13. Buckendahl, C. W., Smith, R. W., Impara, J. C., & Plake, B. S. (2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational measurement, 39(3), 253-263. https://doi.org/10.1111/j.1745-3984.2002.tb01177.x
14. Busch, J.C., & Jaeger, R.M. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the National Teacher Examinations. Journal of Educational Measurement, 27, 145-163. https://doi.org/10.1111/j.1745-3984.1990.tb00739.x
15. Chang, L. (1999). Judgmental item analysis of the Nedelsky and Angoff standard-setting methods. Applied Measurement in Education, 12(2), 151-165. https://doi.org/10.1207/s15324818ame1202_3
16. Çetin, S., & Gelbal, S. (2013). A comparison of Bookmark and Angoff standard setting methods. Educational Sciences: Theory and Practice, 13(4), 2169-2175. DOI: 10.12738/estp.2013.4.1829
17. Chernyshenko, O. S., Stark, S., Drasgow, F., & Roberts, B. W. (2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19(1), 88–106. https://doi.org/10.1037/1040-3590.19.1.88
18. Cizek, G. J. (1996). Setting passing scores. Educational Measurement: Issues and Practice, 15(1), 13-21. https://doi.org/10.1111/j.1745-3992.1996.tb00802.x
19. Cizek, G. J. (2001). Setting performance standards. Lawrence Erlbaum Associates.
20. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on test. Sage.
21. Clauser, B. E., Margolis, M. J., & Case, S. M. (2006). Testing for licensure and certification in the professions. In R.L. Brennan (Ed.) Educational measurement (4th ed., pp. 701-731). Praeger Publishers.
22. Cohen J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. https://doi.org/10.1177/00131644600200010
23. de Ayala, R. J. (2009). The theory and practice of item response theory. Guilford Press.
24. DeMars, C. (2010). Item response theory. Oxford University Press.
25. Dimitrov, D. M. (2023). D-scoing method of measurement: Classical and latent frameworks. Routledge.
26. Dimitrov, D. M. (2022). The response vector for mastery (RVM) method of standard setting. Educational and Psychological Measurement, 82(4), 719-746. https://doi.org/10.1177/00131644211032388
27. Dimitrov, D. M. (2012). Statistical methods for validation of assessment scales in counseling and related fields. Wiley.
28. Dimitrov, D. M., Ghamdi, H., & Alahmadi, M. (2020, December). Setting cutting scores for mastery on Mawhibah’s multiple cognitive ability tests (Technical Report 12-2020), Riyadh, Saudi Arabia: Education & Training Evaluation Commission.
29. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
30. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7(1), 1-26. https://doi.org/10.1214/aos/1176344552
31. Egan, K. L., Ferrara, S., Schneider, M. C., & Barton, K. E. (2009). Writing performance level descriptors and setting performance standards for assessments of modified achievement standards: The role of innovation and importance of following conventional practice. Peabody Journal of Education, 84(4), 552-577. https://doi.org/10.1080/01619560903241028
32. Ferrara, S., Perie, M., & Johnson, E. (2008). Matching the judgmental task with standard setting panelist expertise: The item-descriptor (ID) matching procedure. Journal of Applied Testing Technology, 9(1), 1-22.
33. Ferrara, S., Swaffield, S., & Mueller, L. (2009). Conceptualizing and setting performance standards for alternate assessments. In W.D. Schafer & R.W. Lissitz (Eds.), Alternate assessments based on alternate achievement standards: Policy, practice, and potential (pp. 93–111). Brookes.
34. Gündeğer, C., & Doğan, N. (2018). A statistical approach to improve the standard setting process: Cluster analysis. Journal of Social Sciences, 8(2), 387-397.
35. Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R.L. Brennan (Ed.), Educational measurement (4th ed.). American Council on Education/Praeger Publishers.
36. Hambleton, R. K, Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage.
37. Hambleton, R. K., & Plake, B. S. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education 8(1), 41–55. https://doi.org/10.1207/s15324818ame0801_4
38. Han, C. (2012). Fixing the c parameter in the three-parameter logistic model. Practical Assessment, Research & Evaluation, 17(1), 1-24. https://doi.org/10.7275/f0gz-kc87
39. Hess, B., Subhiyah, R. G., & Giordano, C. (2007). Convergence between cluster analysis and the Angoff method for setting minimum passing scores on credentialing examinations. Evaluation & The Health Professions, 30(4), 362-375. https://doi.org/10.1177/0163278707307904
40. Hu, L. T., & Bentler, P. M. (1999). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424-453. https://doi.org/10.1037/1082-989X.3.4.424
41. Jaeger, R. M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 485–514). Macmillan.
42. Jalili, M., Hejri, S. M., & Norcini, J. J. (2011). Comparison of two methods of standard setting: The performance of the three‐level Angoff method. Medical Education, 45(12), 1199-1208. https://doi.org/10.1111/j.1365-2923.2011.04073.x
43. Kane, M. T. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425–461. https://doi.org/10.3102/00346543064003425
44. Kane, M. T. (2001). So much remains the same: Conception and status of validation in setting standards. In G.J. Cizek (Ed.), Setting performance standards. Lawrence Erlbaum Associates.
45. Kara, H., & Cetin, S. (2020). Comparison of passing scores determined by the Angoff method in different item samples. International Journal of Assessment Tools in Education, 7(1), 80-97. https://doi.org/10.21449/ijate.699479
46. Karantonis, A., & Sireci, S. G. (2006). The bookmark standard‐setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4-12. https://doi.org/10.1111/j.1745-3992.2006.00047.x
47. Keuning, J., Straat, J. H., & Feskens, R. C. W. (2017). The data-driven direct consensus (3DC) procedure: A new approach to standard setting. In S. Blömeke & J. E. Gustafsson (Eds.), Standard setting in education (pp. 287–305). Springer. https://doi.org/10.1007/978-3-319-50856-6_15
48. Khalid, M. N. (2011). Cluster analysis: A standard setting technique in measurement and testing. Journal of Applied Quantitative Methods, 6(2), 46-58.
49. Kline, R. B. (2023). Principles and practice of structural equation modeling (5th ed.). Guilford Press.
50. Lathrop, Q. N. (2014). R package cacIRT: Estimation of classification accuracy and consistency under item response theory. Applied Psychological Measurement, 38(7), 581-582. doi:10.1177/0146621614536465
51. Lee, W. (2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47(1), 1-17.
52. Lewis, D. M., Mitzel, H. C., Green, D. R., & Patz, R. J. (1999). The bookmark standard setting procedure. McGraw-Hill.
53. Lewis, D. M., Mitzel, H. C., Mercado, R. L., & Schulz, E. M. (2012). The bookmark standard setting procedure. In G.J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 225-254). Routledge.
54. Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2001). The bookmark procedure: Psychological perspectives. In G.J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 249–281). Lawrence Erlbaum.
55. Muthén, L. K., & Muthén, B. O. (2024). Mplus (Version 8.11) [Computer software]. Muthén & Muthén.
56. Özarkan, H. B., & Dogan, C. D. (2020). A comparison of two standard-setting methods for tests consisting of constructed-response items. Eurasian Journal of Educational Research, 90, 121-138. https://doi.org/10.14689/ejer.2020.90.7
57. Peabody, M. R., & Wind, S. A. (2019). Exploring the influence of judge proficiency on standard‐setting judgments. Journal of Educational Measurement, 56(1), 101-120. https://doi.org/10.1111/jedm.12202
58. Phillips, G. W. (2012). The benchmark method of standard setting. In G.J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 323-346). Routledge.
59. Plake, B. S., Cizek, G. J. (2012). Variations on a theme: The modified Angoff, extended Angoff, and Yes/No standard setting methods. In G.J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 181-200). Routledge.
60. Raykov, T. (2009). Evaluation of scale reliability for unidimensional measures using latent variable modeling. Measurement and Evaluation in Counseling and Development, 42(3), 223-232. https://doi.org/10.1177/0748175609344096
61. Revelle, W. (2025). psych: Procedures for Psychological, Psychometric, and Personality Research (Version 2.5.6) [Computer software]. Northwestern University. https://CRAN.R-project.org/package=psych
62. Ricker, K. L. (2006). Setting cut-scores: A critical review of the Angoff and modified Angoff methods. The Alberta Journal of Educational Research, 52(1), 53-64. https://doi.org/10.55016/ojs/ajer.v52i1.55111
63. Roberts, J. S. (2016). Generalized Graded Unfolding Model. In W. J. van der Linden (Ed.), Handbook of item response theory, volume one: Models (pp.369-390). (Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences). Chapman & Hall/CRC.
64. Rudner, L. M. (2001). Computing the expected proportions of misclassified examinees. Practical Assessment, Research & Evaluation, 7(14), 1-5. https://doi.org/10.7275/an9m-2035
65. Rudner, L. M. (2005). Expected classification accuracy. Practical Assessment, Research & Evaluation, 10(13), 1-4. https://doi.org/10.7275/56a5-6b14
66. Sayar, A., Dimitrov, D., & Sadaawi, A. (2024, August). Standard-Setting Procedures for the National Assessment for Schools in Saudi Arabia. Paper presented at the 2024 Annual Meeting of the American Psychological Association, Seatle, WA.
67. Sireci, S. G., Robin, F., & Patelis, T. (1999). Using cluster analysis to facilitate standard setting. Applied Measurement in Education, 12(3), 301-325. https://doi.org/10.1207/S15324818AME1203_5
68. Skorupski, W. P., & Hambleton, R. K. (2005). What are panelists thinking when they participate in standard-setting studies? Applied Measurement in Education, 18(3), 233–256. https://doi.org/10.1207/s15324818ame1803_3
69. Strickman, S. N. (2017). You only live up to the standards you set: An evaluation of different approaches to standard setting (Doctoral dissertation) James Madison University.
70. Thomas, C. L., Cassady, J. C., & Finch, W. H. (2018). Identifying severity standards on the cognitive test anxiety scale: Cut score determination using latent class and cluster analysis. Journal of Psychoeducational Assessment, 36(5), 492-508. https://doi.org/10.1177/0734282916686004
71. Weiss, D. J., & Minden, S. V. (2012). A comparison of item parameter estimates from Xcalibre 4.1 and Bilog-MG (Technical Report). Assessment Systems Corporation.
72. Wyse, A., Bunch, M., Deville, C., & Viger, S. G. (2014). A body of work standard-setting method with construct maps. Educational and Psychological Measurement, 74(2), 236-262. https://doi.org/10.1177/0013164413502037
73. Wyse, A. E., & Hao, S. (2012). An evaluation of item response theory classification accuracy and consistency indices. Applied Psychological Measurement, 36(7), 602- 624. https://doi.org/10.1177/0146621612451522
74. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125-145. https://doi.org/10.1177/014662168400800201
75. Yim, M. (2018). Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination. Korean Journal of Medical Education, 30(4), 347-357. https://doi.org/10.3946/kjme.2018.110
76. Yousefi, A. M. (2021). Comparison of the validity of bookmark and Angoff standard setting methods in medical performance tests. BMC Medical Education, 21, 1. https://doi.org/10.1186/s12909-020-02436-3
77. Zieky, M. J. (2001). So much has changed: How the setting of cutscores has evolved since the 1980s. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 19-51). Lawrence Erlbaum.
78. Zwick, R., Senturk, D., Wang, J., & Loomis, S. C. (2001). An investigation of alternative methods for item mapping in the National Assessment of Educational Progress. Educational Measurement: Issues and Practice, 20(2), 15–25. https://doi.org/10.1111/j.1745-3992.2001.tb00059.x