Research Article
BibTex RIS Cite

Gaining a Better Understanding of General Mattering Scale: An Application of Classical Test Theory and Item Response Theory

Year 2018, Volume: 5 Issue: 4, 668 - 681, 16.12.2018

Halil İbrahim Sarı Mehmet Akif Karaman

https://doi.org/10.21449/ijate.453337
Cited By: 4

Abstract

The current study shows applications of both classical test theory (CTT) and item response theory (IRT) to psychology data. The study discusses item level analyses of General Mattering Scale produced by two theories as well as strengths and weaknesses of both measurement approaches. The survey consisted of a total of five Likert-type items. Each student chose the best answers from the given categories (e.g., not at all, a little, somewhat, very much). We specifically run generalized partial credit IRT model. Overall, we discussed that while CTT provides comparatively superficial information but, IRT allows gaining deeper insight into the test items and response categories. We concluded that the meaning of item properties differ in CTT and IRT, and this difference may lead to different interpretations. We aimed to encourage psychologists and counsellors to give more consideration to the IRT applications when assessing the psychometric features of the items. 

Keywords

Classical test theory Item response theory Generalized partial model Item analysis General mattering scale

References

  • Arbuckle, J. L. (2014). Amos 23.0 user's guide. Chicago, IL: IBM SPSS.
  • Baker F. (1992). Item response theory. New York, NY: Marcel Dekker, INC.
  • Brodin, U., Fors, U., & Laksov, K. B. (2010). The application of item response theory on a teaching strategy profile questionnaire. BMC medical education, 10, 14-14- doi:10.1186/1472-6920-10-14
  • Cha, M. (2016). The mediation effect of mattering and self-esteem in the relationship between socially prescribed perfectionism and depression: Based on the social disconnection model. Personality and Individual Differences, 88, 148-159. doi:10.1016/j.paid.2015.09.008
  • Cho, S., Drasgow, F., & Cao, M. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27, 1241–1252. doi:10.1037/pas0000132
  • Connolly, K. M., & Myers, J. E. (2003). Wellness and mattering: The role of holistic factors in job satisfaction. Journal of Employment Counseling, 40, 152-160. doi:10.1002/j.2161-1920.2003.tb00866.x
  • Crocker, L. M., & Algina, J. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart and Winston.de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford.
  • Demir, M., Özen, A. & Doğan, A. (2012). Friendship, perceived mattering and happiness: A study of American and Turkish college students. The Journal of Social Psychology, 152, 659-664. doi: 10.1080/00224545.2011.650237
  • Dodeen, H., & Al-Darmaki, F. (2016). The application of item response theory in developing and validating a shortened version of the Emirate Marital Satisfaction Scale. Psychological Assessment, 28, 1625-1633. doi:10.1037/pas0000296.supp
  • Embretson S.E, & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, IN: Lawrence Erlbaum Associates, Inc.
  • Flett, G. L., Galfi-Pechenkov, I., Molnar, D. S., Hewitt, P. L., & Goldstein, A. L. (2012). Perfectionism, mattering, and depression: A mediational analysis. Personality and Individual Differences, 52, 828-832. doi:10.1016/j.paid.2011.12.041
  • Gomez, R., & Vance, A. (2015). Item response theory properties of the internalizing disorders in adolescents. Journal of Childhood & Developmental Disorders, 1, 1-10. doi:10.4172/2472-1786.100006
  • Haktanir, A., Lenz, A. S., Can, N., & Watson, J. C. (2016). Development and evaluation of Turkish language versions of three positive psychology assessments. International Journal for the Advancement of Counselling, 38, 286-297. doi:10.1007/s10447-016-9272-9
  • Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38, II28-II42. doi:10.1097/00005650-200009002-00007
  • Huggins-Manley, A. C., & Algina, J. (2015). The partial credit model and generalized partial credit model as constrained nominal response models, with applications in M Plus. Structural Equation Modeling: A Multidisciplinary Journal, 22, 308-318. doi:10.1080/10705511.2014.937374
  • IBM Corp. (2013). IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp.
  • Karaman, M. A., Balkin, R. S., & Juhnke, G. A. (2018). Turkish adaptation of the Juhnke–Balkin Life Balance Inventory. Measurement and Evaluation in Counseling and Development, 51, 141–150. doi:10.1080/07481756.2017.1308226
  • Lee, Y. S., Krishnan, A., & Park, Y. S. (2012). Psychometric properties of the Children’s Depression Inventory: An item response theory analysis across age in a nonclinical, longitudinal, adolescent sample. Measurement and Evaluation in Counseling and Development, 45, 84-100. doi: 10.1177/0748175611428329
  • Locke, B. D., McAleavey, A. A., Zhao, Y., Lei, P. W., Hayes, J. A., Castonguay, L. G., ... & Lin, Y. C. (2012). Development and initial validation of the Counseling Center Assessment of Psychological Symptoms–34. Measurement and Evaluation in Counseling and Development, 45, 151-169. doi: 10.1177/0748175611432642
  • Lord, F.M. (1980). Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Lawrence Erlbaum.
  • Marcus, F. M. (1991). Mattering: Its measurement and theoretical significance for social psychology. Paper presented at the annual meeting of the Eastern Sociological Association, Cincinnati.
  • Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.
  • Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
  • Rayle, A. D. (2006). Mattering to others: Implications for the counseling relationship. Journal of Counseling & Development, 84, 483-487. doi:10.1002/j.1556-6678.2006.tb00432.x
  • R Core Team (2016). R: A language and environment for statistical computing [computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
  • Riemer, M., & Kearns, M. A. (2010). Description and psychometric evaluation of the Youth Counseling Impact Scale. Psychological Assessment, 22, 259-268. doi:10.1037/a0018507.
  • Rizopoulos, D. (2017). Package “ltm”, Latent Trait Models under IRT. Retrieved from https://cran.r-project.org/web/packages/ltm/ltm.pdf
  • Rosenberg, M., & McCullough, B. C. (1981). Mattering: Inferred significance and mental health among adolescents. Research in Community & Mental Health, 2, 163-182.
  • Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34, 1–97. doi:10.1007/bf03372160 Tasca, G. A., Cabrera, C., Kristjansson, E., MacNair-Semands, R., Joyce, A. S., & Ogrodniczuk, J. S. (2016). The therapeutic factor inventory-8: Using item response theory to create a brief scale for continuous process monitoring for group psychotherapy. Psychotherapy Research, 26, 131-145. doi:10.1080/10503307.2014.963729
  • Tovar, E., Simon, M. A., & Lee, H. B. (2009). Development and validation of the college mattering inventory with diverse urban college students. Measurement and Evaluation in Counseling and Development, 42, 154-178. doi:10.1177/0748175609344091
  • Tovar, E. (2013). A conceptual model on the impact of mattering, sense of belonging, engagement/involvement, and socio-academic integrative experiences on community college students’ intent to persist (Doctoral Dissertation). Retrieved from CGU Theses & Dissertations (Paper 81). doi: 10.5642/cguetd/81
  • Turner, B. M., Betz, N. E., Edwards, M. C., & Borgen, F. H. (2010). Psychometric examination of an inventory of self-efficacy for the Holland vocational themes using item response theory. Measurement and Evaluation in Counseling and Development, 43, 188-198. doi:10.1177/0748175610384810
  • Zanon, C., Hutz, C. S., Yoo, H. H., & Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29, 1-10. doi:10.1186/s41155-016-0040-x

Gaining a Better Understanding of General Mattering Scale: An Application of Classical Test Theory and Item Response Theory

Year 2018, Volume: 5 Issue: 4, 668 - 681, 16.12.2018

Halil İbrahim Sarı Mehmet Akif Karaman

https://doi.org/10.21449/ijate.453337
Cited By: 4

Abstract

The current study shows applications of both classical test theory (CTT) and item response theory (IRT) to psychology data. The study discusses item level analyses of General Mattering Scale produced by two theories as well as strengths and weaknesses of both measurement approaches. The survey consisted of a total of five Likert-type items. Each student chose the best answers from the given categories (e.g., not at all, a little, somewhat, very much). We specifically run generalized partial credit IRT model. Overall, we discussed that while CTT provides comparatively superficial information but, IRT allows gaining deeper insight into the test items and response categories. We concluded that the meaning of item properties differ in CTT and IRT, and this difference may lead to different interpretations. We aimed to encourage psychologists and counsellors to give more consideration to the IRT applications when assessing the psychometric features of the items. 

Keywords

Classical test theory item response theory generalized partial model item analysis general mattering scale

References

  • Arbuckle, J. L. (2014). Amos 23.0 user's guide. Chicago, IL: IBM SPSS.
  • Baker F. (1992). Item response theory. New York, NY: Marcel Dekker, INC.
  • Brodin, U., Fors, U., & Laksov, K. B. (2010). The application of item response theory on a teaching strategy profile questionnaire. BMC medical education, 10, 14-14- doi:10.1186/1472-6920-10-14
  • Cha, M. (2016). The mediation effect of mattering and self-esteem in the relationship between socially prescribed perfectionism and depression: Based on the social disconnection model. Personality and Individual Differences, 88, 148-159. doi:10.1016/j.paid.2015.09.008
  • Cho, S., Drasgow, F., & Cao, M. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27, 1241–1252. doi:10.1037/pas0000132
  • Connolly, K. M., & Myers, J. E. (2003). Wellness and mattering: The role of holistic factors in job satisfaction. Journal of Employment Counseling, 40, 152-160. doi:10.1002/j.2161-1920.2003.tb00866.x
  • Crocker, L. M., & Algina, J. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart and Winston.de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford.
  • Demir, M., Özen, A. & Doğan, A. (2012). Friendship, perceived mattering and happiness: A study of American and Turkish college students. The Journal of Social Psychology, 152, 659-664. doi: 10.1080/00224545.2011.650237
  • Dodeen, H., & Al-Darmaki, F. (2016). The application of item response theory in developing and validating a shortened version of the Emirate Marital Satisfaction Scale. Psychological Assessment, 28, 1625-1633. doi:10.1037/pas0000296.supp
  • Embretson S.E, & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, IN: Lawrence Erlbaum Associates, Inc.
  • Flett, G. L., Galfi-Pechenkov, I., Molnar, D. S., Hewitt, P. L., & Goldstein, A. L. (2012). Perfectionism, mattering, and depression: A mediational analysis. Personality and Individual Differences, 52, 828-832. doi:10.1016/j.paid.2011.12.041
  • Gomez, R., & Vance, A. (2015). Item response theory properties of the internalizing disorders in adolescents. Journal of Childhood & Developmental Disorders, 1, 1-10. doi:10.4172/2472-1786.100006
  • Haktanir, A., Lenz, A. S., Can, N., & Watson, J. C. (2016). Development and evaluation of Turkish language versions of three positive psychology assessments. International Journal for the Advancement of Counselling, 38, 286-297. doi:10.1007/s10447-016-9272-9
  • Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38, II28-II42. doi:10.1097/00005650-200009002-00007
  • Huggins-Manley, A. C., & Algina, J. (2015). The partial credit model and generalized partial credit model as constrained nominal response models, with applications in M Plus. Structural Equation Modeling: A Multidisciplinary Journal, 22, 308-318. doi:10.1080/10705511.2014.937374
  • IBM Corp. (2013). IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp.
  • Karaman, M. A., Balkin, R. S., & Juhnke, G. A. (2018). Turkish adaptation of the Juhnke–Balkin Life Balance Inventory. Measurement and Evaluation in Counseling and Development, 51, 141–150. doi:10.1080/07481756.2017.1308226
  • Lee, Y. S., Krishnan, A., & Park, Y. S. (2012). Psychometric properties of the Children’s Depression Inventory: An item response theory analysis across age in a nonclinical, longitudinal, adolescent sample. Measurement and Evaluation in Counseling and Development, 45, 84-100. doi: 10.1177/0748175611428329
  • Locke, B. D., McAleavey, A. A., Zhao, Y., Lei, P. W., Hayes, J. A., Castonguay, L. G., ... & Lin, Y. C. (2012). Development and initial validation of the Counseling Center Assessment of Psychological Symptoms–34. Measurement and Evaluation in Counseling and Development, 45, 151-169. doi: 10.1177/0748175611432642
  • Lord, F.M. (1980). Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Lawrence Erlbaum.
  • Marcus, F. M. (1991). Mattering: Its measurement and theoretical significance for social psychology. Paper presented at the annual meeting of the Eastern Sociological Association, Cincinnati.
  • Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.
  • Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
  • Rayle, A. D. (2006). Mattering to others: Implications for the counseling relationship. Journal of Counseling & Development, 84, 483-487. doi:10.1002/j.1556-6678.2006.tb00432.x
  • R Core Team (2016). R: A language and environment for statistical computing [computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
  • Riemer, M., & Kearns, M. A. (2010). Description and psychometric evaluation of the Youth Counseling Impact Scale. Psychological Assessment, 22, 259-268. doi:10.1037/a0018507.
  • Rizopoulos, D. (2017). Package “ltm”, Latent Trait Models under IRT. Retrieved from https://cran.r-project.org/web/packages/ltm/ltm.pdf
  • Rosenberg, M., & McCullough, B. C. (1981). Mattering: Inferred significance and mental health among adolescents. Research in Community & Mental Health, 2, 163-182.
  • Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34, 1–97. doi:10.1007/bf03372160 Tasca, G. A., Cabrera, C., Kristjansson, E., MacNair-Semands, R., Joyce, A. S., & Ogrodniczuk, J. S. (2016). The therapeutic factor inventory-8: Using item response theory to create a brief scale for continuous process monitoring for group psychotherapy. Psychotherapy Research, 26, 131-145. doi:10.1080/10503307.2014.963729
  • Tovar, E., Simon, M. A., & Lee, H. B. (2009). Development and validation of the college mattering inventory with diverse urban college students. Measurement and Evaluation in Counseling and Development, 42, 154-178. doi:10.1177/0748175609344091
  • Tovar, E. (2013). A conceptual model on the impact of mattering, sense of belonging, engagement/involvement, and socio-academic integrative experiences on community college students’ intent to persist (Doctoral Dissertation). Retrieved from CGU Theses & Dissertations (Paper 81). doi: 10.5642/cguetd/81
  • Turner, B. M., Betz, N. E., Edwards, M. C., & Borgen, F. H. (2010). Psychometric examination of an inventory of self-efficacy for the Holland vocational themes using item response theory. Measurement and Evaluation in Counseling and Development, 43, 188-198. doi:10.1177/0748175610384810
  • Zanon, C., Hutz, C. S., Yoo, H. H., & Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29, 1-10. doi:10.1186/s41155-016-0040-x
There are 33 citations in total.

Details

Primary Language English
Subjects Studies on Education
Journal Section Articles
Authors

Halil İbrahim Sarı 0000-0001-7506-9000

Mehmet Akif Karaman

Publication Date December 16, 2018
Submission Date August 14, 2018
Published in Issue Year 2018 Volume: 5 Issue: 4

Cite

APA Sarı, H. İ., & Karaman, M. A. (2018). Gaining a Better Understanding of General Mattering Scale: An Application of Classical Test Theory and Item Response Theory. International Journal of Assessment Tools in Education, 5(4), 668-681. https://doi.org/10.21449/ijate.453337

Cited By

Investigation of education value perception scale's psychometric properties according to CTT and IRT
International Journal of Assessment Tools in Education
https://doi.org/10.21449/ijate.986530
Perceptions of Accountability to God and Psychological Well-Being Among US Adults
Journal of Religion and Health
https://doi.org/10.1007/s10943-021-01471-8
Development of Perceived School Counselor Support Scale: Based on the ASCA Mindsets and Behaviors
International Journal of Assessment Tools in Education
Mehmet Akif Karaman
https://doi.org/10.21449/ijate.561584
The Interpersonal Mattering Scale: its reliability and validity in an Iranian sample
Mental Health, Religion & Culture
Mahboubeh Dadfar
https://doi.org/10.1080/13674676.2020.1726884

23824         23823             23825