Use of Item Response Theory to Validate Cyberbullying Sensibility Scale for University Students

Abstract

A thirteen-item cyberbullying sensibility scale (CSS), developed by Tanrıkulu, Kınay, and Arıcak (2013) and extensively used by researchers, was used to measure the cyberbullying sensibility levels of high school students. Unlike other similar concepts, such as cyberbullying and cyber victimization, there are no scales developed to measure the cyberbullying sensibility among university students. In this study, the data obtained from 727 university students were analyzed based on item response theory (IRT) techniques, and psychometric evidences were obtained to evaluate whether it is appropriate to use the scale on the university students. Accordingly, a parameterization of CSS items was performed by using the graded response model. Using the discrimination parameters and item fit statistics, some items were removed from the original scale and a seven-item CSS version was developed since preliminary exploratory and confirmatory factor analyses provide inadequate evidence for the validity of a one-dimensional structure of cyberbullying sensibility. However, an IRT-based item removal process yielded an acceptable improvement. In this way, despite the six items being removed from the original CSS form, the scale retained 64% of the information it provided. The reliability values computed based on the classical approach and IRT were above .8 after the item elimination process with only a minor drop. With the validation process, the CSS will be a valuable measurement tool to determine the level of cyberbullying sensibility among university students and allow academicians to conduct research with this population.

Keywords

• cyberbullying sensibility
• test validation
• Item response theory
• Graded response model
• Item selection

Use of Item Response Theory to Validate Cyberbullying Sensibility Scale for University Students

Öz

A thirteen-item cyberbullying sensibility scale (CSS), developed by Tanrıkulu, Kınay, and Arıcak (2013) and extensively used by researchers, was used to measure the cyberbullying sensibility levels of high school students. Unlike other similar concepts, such as cyberbullying and cyber victimization, there are no scales developed to measure the cyberbullying sensibility among university students. In this study, the data obtained from 727 university students were analyzed based on item response theory (IRT) techniques, and psychometric evidences were obtained to evaluate whether it is appropriate to use the scale on the university students. Accordingly, a parameterization of CSS items was performed by using the graded response model. Using the discrimination parameters and item fit statistics, some items were removed from the original scale and a seven-item CSS version was developed since preliminary exploratory and confirmatory factor analyses provide inadequate evidence for the validity of a one-dimensional structure of cyberbullying sensibility. However, an IRT-based item removal process yielded an acceptable improvement. In this way, despite the six items being removed from the original CSS form, the scale retained 64% of the information it provided. The reliability values computed based on the classical approach and IRT were above .8 after the item elimination process with only a minor drop. With the validation process, the CSS will be a valuable measurement tool to determine the level of cyberbullying sensibility among university students and allow academicians to conduct research with this population.

Anahtar Kelimeler

• Cyberbullying sensibility
• Test validation
• Item response theory
• Graded response model
• Item selection

References

1. Álvarez-García, D., Núñez, J.C., González-Castro, P., Rodríguez, C., & Cerezo, R. (2019) The Effect of Parental Control on Cyber-Victimization in Adolescence: The Mediating Role of Impulsivity and High-Risk Behaviors. Front. Psychol., 10, 1159.
2. Ang, R.P., & Goh, D.H. (2010). Cyberbullying among adolescents: The role of affective and cognitive empathy, and gender. Child Psychiatry & Human Development, 41(4), 387-397.
3. Bilker, W.B., Hansen, J. A., Brensinger, C. M., Richard, J., Gur, R. E., & Gur, R. C. (2012). Development of abbreviated nine-item forms of the Raven's standard progressive matrices test. Assessment, 19(3), 354-369.
4. Baker, F.B. (2001). The basics of item response theory (2nd ed.). College Park, MD: ERIC Clearinghouse on Assessment and Evaluation, University of Maryland. Retrieved February, 3 2019 from http://files.eric.ed.gov/fulltext/ED458219.pdf
5. Baştak, G., & Altınova, H.H. (2015). Lise Öğrencilerinde Yaratıcı Drama Yöntemiyle Siber Zorbalık Hakkında Duyarlılık Oluşturma. Yaratıcı Drama Dergisi, 10(1), 91-102.
6. Chalmers, R.P. (2012). mirt: A Multidimensional Item Response Theory Package for the R Environment. Journal of Statistical Software, 48(6), 1-29.
7. Doğan, E., Cansu, Ç., & Şahin, Y.L. (2016). A Study on Online Social Network Games Players’ Cyberbullying Sensibility and Aims of Facebook Usage/Çevrimiçi Sosyal Ağ Oyunu Oynayan Bireylerin Siber Zorbaliğa Duyarlılık Düzeyleri ile Facebook Kullanım Amaçları Üzerine Bir Çalışma. Eğitimde Kuram ve Uygulama, 12(3), 501-520.
8. Embretson, S., Reise, S.P. (2000). Item Response Theory for Psychologists. Lawrence Erlbaum Associates. Inc. Mahwah.

9. Gaffney, H., Farrington, D. P., Espelage, D. L., and Ttofi, M. M. (2019). Are cyberbullying intervention and prevention programs effective? a systematic and meta-analytical review. Aggress. Violent Behav. 45, 134–153. doi: 10.1016/j.avb.2018.07.002
10. Gahagan, K., Vaterlaus, J.M., & Frost, L.R. (2016). College student cyberbullying on social networking sites: Conceptualization. prevalence. and perceived bystander responsibility. Computers in human behavior, 55, 1097-1105.
11. Hambleton, R.K, Swaminathan, H., Rogers, H.J. (1991). Fundamentals of Item Response Theory. Thousand Oaks: Sage Publications.
12. Hambleton, R.K., Robin, F., & Xing, D. (2000). Item response models for the analysis of educational and psychological test data. In: Tinsley HEA. Brown SD. editors. Handbook of Applied Multivariate Statistics and Mathematical Modeling. San Diego: Academic. p. 553–85.
13. Hambleton, R.K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer Academic Publishers.
14. Hattie, J. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139–64.
15. Hu. L. T. & Bentler. P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
16. Istiyono, E., Dwandaru, W.S.B., Lede, Y.A., Rahayu, F., & Nadapdap, A. (2019). Developing IRT-Based Physics Critical Thinking Skill Test: A CAT to Answer 21st Century Challenge. International Journal of Instruction, 12(4), 267-280.
17. Kokkinos, C.M., Antoniadou, N., & Markos, A. (2014). Cyber-bullying: An investigation of the psychological profile of university student participants. Journal of Applied Developmental Psychology, 35(3), 204-214.
18. Lee, J., Abell, N., & Holmes, J.L. (2015). Validation of measures of cyberbullying perpetration and victimization in emerging adulthood. Research on Social Work Practice. http://dx.doi.org/10.1177/1049731515578535
19. Mielenz, T.J., Edwards, M.C. & Callahan, L.F. (2010). Item response theory analysis of two questionnaire measures of arthritis-related self efficacy beliefs from community based US samples. Hindawi Publishing Corporation Arthritis.
20. Muthén, L.K., & Muthén, B.O. (1998-2012). Mplus User’s Guide: Statistical Analysis with Latent Variables (7th ed.). Los Angeles. CA: Muthén & Muthén.
21. Nedim-Bal, P., & Kahraman, S. (2015). The Effect of Cyber Bullying Sensibility Improvement Group Training Program on Gifted Students. Journal of Gifted Education Research, 3(2). 48-57.
22. Ojedokun, O., & Idemudia, E. S. (2013). The moderating role of emotional intelligence between PEN personality factors and cyberbullying in a student population. Life Science Journal, 10(3), 1924-1930.
23. Orlando, M., & Thissen, D. (2000). New item fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50-64.
24. R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna. Austria. URL https://www.R-project.org/
25. Rubio, V.J., Aguado, D., Hontangas, P.M., & Hernandez, J.M. (2007). Psychometric properties of an emotional adjustment measure. European Journal of Psychological Assessment, 23 (1), 39-46.
26. Samejima, F. (1969) Estimation of Latent Ability Using a Response Pattern of Graded Scores. (Psychometrika Monograph, No. 17). Psychometric Society, Richmond. http://www.psychometrika.org/journal/online/MN17.pdf
27. Tanrıkulu, I. (2018). Cyberbullying prevention and intervention programs in schools: A systematic review. School psychology international, 39(1), 74-91.
28. Tanrıkulu, T., Kınay, H. & Arıcak, O.T. (2013). Cyberbullying sensibility scale: validity and reliability study. Trakya University Journal of Education, 3(1), 38-47.
29. Tanrıkulu, T., Kınay, H., & Arıcak. O. T. (2015). Sensibility development program against cyberbullying. New Media & Society, 17(5), 708-719.
30. Thissen, D., & Steinberg, L. (1986). A taxonomy of item response models. Psychometrika, 51(4), 567–577.
31. Tolia, A. (2016). Cyberbullying: Psychological effect on children. The International Journal of Indian Psychology, 3(2), No. 1. 48-51.
32. Tsutakawa, R.K., & Johnson. J.C. (1990). The effect of uncertainty of item parameter estimation on ability estimates. Psychometrika, 55, 371–390.
33. Willoughby, M. (2018). A review of the risks associated with children and young people’s social media use and the implications for social work practice. Journal of Social Work Practice. 1-14. doi:10.1080/02650533.2018.1460587
34. Yen, W.M. (1984). Effects of local item dependence on the fit and equating performance of the three parameter logistic model. Applied Psychological Measurement, 8, 125-145.
35. Zanon, C., Hutz, C.S., Yoo, H., & Hambleton, R.K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29, 1-10.