Gaining a Better Understanding of General Mattering Scale: An Application of Classical Test Theory and Item Response Theory

Abstract

The current study shows applications of both classical test theory (CTT) and item response theory (IRT) to psychology data. The study discusses item level analyses of General Mattering Scale produced by two theories as well as strengths and weaknesses of both measurement approaches. The survey consisted of a total of five Likert-type items. Each student chose the best answers from the given categories (e.g., not at all, a little, somewhat, very much). We specifically run generalized partial credit IRT model. Overall, we discussed that while CTT provides comparatively superficial information but, IRT allows gaining deeper insight into the test items and response categories. We concluded that the meaning of item properties differ in CTT and IRT, and this difference may lead to different interpretations. We aimed to encourage psychologists and counsellors to give more consideration to the IRT applications when assessing the psychometric features of the items. 

Keywords

• Classical test theory
• Item response theory
• Generalized partial model
• Item analysis
• General mattering scale

References

1. Arbuckle, J. L. (2014). Amos 23.0 user's guide. Chicago, IL: IBM SPSS.
2. Baker F. (1992). Item response theory. New York, NY: Marcel Dekker, INC.
3. Brodin, U., Fors, U., & Laksov, K. B. (2010). The application of item response theory on a teaching strategy profile questionnaire. BMC medical education, 10, 14-14- doi:10.1186/1472-6920-10-14
4. Cha, M. (2016). The mediation effect of mattering and self-esteem in the relationship between socially prescribed perfectionism and depression: Based on the social disconnection model. Personality and Individual Differences, 88, 148-159. doi:10.1016/j.paid.2015.09.008
5. Cho, S., Drasgow, F., & Cao, M. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27, 1241–1252. doi:10.1037/pas0000132
6. Connolly, K. M., & Myers, J. E. (2003). Wellness and mattering: The role of holistic factors in job satisfaction. Journal of Employment Counseling, 40, 152-160. doi:10.1002/j.2161-1920.2003.tb00866.x
7. Crocker, L. M., & Algina, J. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart and Winston.de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford.
8. Demir, M., Özen, A. & Doğan, A. (2012). Friendship, perceived mattering and happiness: A study of American and Turkish college students. The Journal of Social Psychology, 152, 659-664. doi: 10.1080/00224545.2011.650237

9. Dodeen, H., & Al-Darmaki, F. (2016). The application of item response theory in developing and validating a shortened version of the Emirate Marital Satisfaction Scale. Psychological Assessment, 28, 1625-1633. doi:10.1037/pas0000296.supp
10. Embretson S.E, & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, IN: Lawrence Erlbaum Associates, Inc.
11. Flett, G. L., Galfi-Pechenkov, I., Molnar, D. S., Hewitt, P. L., & Goldstein, A. L. (2012). Perfectionism, mattering, and depression: A mediational analysis. Personality and Individual Differences, 52, 828-832. doi:10.1016/j.paid.2011.12.041
12. Gomez, R., & Vance, A. (2015). Item response theory properties of the internalizing disorders in adolescents. Journal of Childhood & Developmental Disorders, 1, 1-10. doi:10.4172/2472-1786.100006
13. Haktanir, A., Lenz, A. S., Can, N., & Watson, J. C. (2016). Development and evaluation of Turkish language versions of three positive psychology assessments. International Journal for the Advancement of Counselling, 38, 286-297. doi:10.1007/s10447-016-9272-9
14. Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38, II28-II42. doi:10.1097/00005650-200009002-00007
15. Huggins-Manley, A. C., & Algina, J. (2015). The partial credit model and generalized partial credit model as constrained nominal response models, with applications in M Plus. Structural Equation Modeling: A Multidisciplinary Journal, 22, 308-318. doi:10.1080/10705511.2014.937374
16. IBM Corp. (2013). IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp.
17. Karaman, M. A., Balkin, R. S., & Juhnke, G. A. (2018). Turkish adaptation of the Juhnke–Balkin Life Balance Inventory. Measurement and Evaluation in Counseling and Development, 51, 141–150. doi:10.1080/07481756.2017.1308226
18. Lee, Y. S., Krishnan, A., & Park, Y. S. (2012). Psychometric properties of the Children’s Depression Inventory: An item response theory analysis across age in a nonclinical, longitudinal, adolescent sample. Measurement and Evaluation in Counseling and Development, 45, 84-100. doi: 10.1177/0748175611428329
19. Locke, B. D., McAleavey, A. A., Zhao, Y., Lei, P. W., Hayes, J. A., Castonguay, L. G., ... & Lin, Y. C. (2012). Development and initial validation of the Counseling Center Assessment of Psychological Symptoms–34. Measurement and Evaluation in Counseling and Development, 45, 151-169. doi: 10.1177/0748175611432642
20. Lord, F.M. (1980). Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Lawrence Erlbaum.
21. Marcus, F. M. (1991). Mattering: Its measurement and theoretical significance for social psychology. Paper presented at the annual meeting of the Eastern Sociological Association, Cincinnati.
22. Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.
23. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
24. Rayle, A. D. (2006). Mattering to others: Implications for the counseling relationship. Journal of Counseling & Development, 84, 483-487. doi:10.1002/j.1556-6678.2006.tb00432.x
25. R Core Team (2016). R: A language and environment for statistical computing [computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
26. Riemer, M., & Kearns, M. A. (2010). Description and psychometric evaluation of the Youth Counseling Impact Scale. Psychological Assessment, 22, 259-268. doi:10.1037/a0018507.
27. Rizopoulos, D. (2017). Package “ltm”, Latent Trait Models under IRT. Retrieved from https://cran.r-project.org/web/packages/ltm/ltm.pdf
28. Rosenberg, M., & McCullough, B. C. (1981). Mattering: Inferred significance and mental health among adolescents. Research in Community & Mental Health, 2, 163-182.
29. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34, 1–97. doi:10.1007/bf03372160 Tasca, G. A., Cabrera, C., Kristjansson, E., MacNair-Semands, R., Joyce, A. S., & Ogrodniczuk, J. S. (2016). The therapeutic factor inventory-8: Using item response theory to create a brief scale for continuous process monitoring for group psychotherapy. Psychotherapy Research, 26, 131-145. doi:10.1080/10503307.2014.963729
30. Tovar, E., Simon, M. A., & Lee, H. B. (2009). Development and validation of the college mattering inventory with diverse urban college students. Measurement and Evaluation in Counseling and Development, 42, 154-178. doi:10.1177/0748175609344091
31. Tovar, E. (2013). A conceptual model on the impact of mattering, sense of belonging, engagement/involvement, and socio-academic integrative experiences on community college students’ intent to persist (Doctoral Dissertation). Retrieved from CGU Theses & Dissertations (Paper 81). doi: 10.5642/cguetd/81
32. Turner, B. M., Betz, N. E., Edwards, M. C., & Borgen, F. H. (2010). Psychometric examination of an inventory of self-efficacy for the Holland vocational themes using item response theory. Measurement and Evaluation in Counseling and Development, 43, 188-198. doi:10.1177/0748175610384810
33. Zanon, C., Hutz, C. S., Yoo, H. H., & Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29, 1-10. doi:10.1186/s41155-016-0040-x