Development of Organizational Cybersecurity Awareness Scale (OCAS)

Abstract

This study reports the development and validation of the Organizational Cybersecurity Awareness Scale (OCAS) for employees in public institutions. The scale comprises four dimensions: (1) Personal Awareness and Proactive Approach, (2) Legal and Regulatory Awareness, (3) Technical Protection and Implementation, and (4) Organizational Policy Awareness, with a total of 26 items. Data were collected in 2021 from 110 employees working in two public organizations to evaluate the scale’s EFA analysis. Exploratory Factor Analysis (EFA) was conducted to establish the dimensional structure of the scale. The Kaiser-Meyer-Olkin measure of sampling adequacy was 0.942, indicating excellent factorability. The four-factor solution explained 60.02% of the total variance: Factor 1 (Personal Awareness and Proactive Approach) accounted for 39.54%, Factor 2 (Legal and Regulatory Awareness) for 11.19%, Factor 3 (Technical Protection and Implementation) for 4.76%, and Factor 4 (Organizational Policy Awareness) for 4.53%. All 26 items demonstrated satisfactory factor loadings, and no items required deletion. Confirmatory Factor Analysis (CFA) was performed to test the hypothesized factor structure. The overall scale demonstrated excellent internal consistency (Cronbach’s α = 0.936). Although the chi-square test was significant (χ² = 7952.667, df = 325, N = 544, p < .05), which is common in large samples, other fit indices indicated acceptable model fit: Standardized RMR = 0.085 and RMSEA = 0.083. To test the hypothesized four-factor structure derived from EFA, CFA was performed by using another 544 employees. These results support the four-dimensional structure and demonstrate that the OCAS is a reliable and valid instrument for measuring cybersecurity awareness among public sector employees.

Keywords

• Cybersecurity awareness
• public employees
• scale development
• organizational security
• psychometric validation

References

1. Abawajy, J. (2014). User preference of cyber security awareness delivery methods. Behaviour & Information Technology, 33(3), 237–248.
2. Al-Shanfari, I., Yassin, W., Tabook, N., et al. (2022). Determinants of Information Security Awareness and Behaviour Strategies in Public Sector Organizations among Employees. International Journal of Advanced Computer Science and Applications, 13(8), 855-864. https://doi.org/10.14569/ijacsa.2022.0130855
3. AlMindeel, R., & Martins, J. T. (2020). Information security awareness in a developing country context: Insights from the government sector in Saudi Arabia. Information Technology & People, 34(3), 770-792. https://doi.org/10.1108/ITP-06-2019-0269
4. Almuqrin, A., Mutambik, I., Zhang, Z., et al. (2024). Cracking the Code: Unraveling the Determinants of Cybersecurity Awareness Among University Students. Journal of Organizational and End User Computing, 36(1), 1-25. https://doi.org/10.4018/joeuc.345933
5. Arpaci, I., & Aslan, Ö. (2022). Development of a scale to measure cybercrime-awareness on social media. Journal of Computer Information Systems, 63(3), 695–705. https://doi.org/10.1080/08874417.2022.2101160
6. Arpaci, I., & Sevinc, K. (2022). Development of the cybersecurity scale (CS-S): Evidence of validity and reliability. Information Development, 38(2), 218-226.
7. Aşan, C. (2024). Developing a measurement scale to assess the perception of cybersecurity among employees in the maritime industry. Journal of Naval Science and Engineering, 20(2), 135-162. https://doi.org/10.56850/jnse.1485985
8. Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42(5), 815-824. https://doi.org/10.1016/j.paid.2006.09.018

9. Bognár, F., & Bottyán, L. (2024). Development and Validation of a Personal Cybersecurity Awareness Scale. Sensors, 24(11), 3543. https://doi.org/10.3390/s24113543
10. Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford Publications.
11. Chapman, T. A. (2019). Factors Affecting Perceptions of Cybersecurity Readiness Among Workgroup IT Managers [Doctoral dissertation]. https://www.proquest.com/docview/2303848162
12. Creasey, J. (2013). Cybersecurity incident response guide. https://www.crestapproved.org/wpcontent/uploads/2014/11/CSIRProcurementGuide.pdf
13. DeVellis, R. F. (2017). Scale Development: Theory and Applications (4th ed.). Sage Publications.
14. Di Nocera, F., Ricciardi, O., Longo, Y., Ciani, G., & Ferlazzo, F. (2024). The CAIN Inventory: Measurement Invariance and Construct Validity of a Measure of Cybersecurity Awareness. Cyberpsychology, Behavior, and Social Networking, 27(3), 183-190. https://doi.org/10.1089/cyber.2023.0350
15. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299. https://doi.org/10.1037/1082-989X.4.3.272
16. Fadhil, A., & Yazid, S. (2023). Measurement of Employee Information Security Awareness: A Case Study of National Civil Service Agency. The Indonesian Journal of Computer Science, 12(6).
17. Faklaris, C., Dabbish, L., & Hong, J. I. (2022). SA-13: A Validated Security Attitudes Inventory. ACM Transactions on Computer-Human Interaction, 29(6), Article 51. https://doi.org/10.1145/3532867
18. Fox, R. J. (2010). Confirmatory factor analysis. In Wiley Encyclopedia of Management. https://doi.org/10.1002/9781444316568.WIEM02060
19. Georgiadou, A., Mouzakitis, S., Bounas, K., et al. (2020). A Cyber-Security Culture Framework for Assessing Organization Readiness. Journal of Computer Information Systems, 62(3), 452-462. https://doi.org/10.1080/08874417.2020.1845583
20. Goode, S., Hoehle, H., Venkatesh, V., & Brown, S. A. (2018). User compensation as a data breach recovery action: An investigation of the Sony PlayStation Network breach. MIS Quarterly, 41(3), 703-727. https://doi.org/10.25300/MISQ/2017/41.3.03
21. Güldüren, C., Çetinkaya, L., & Keser, H. (2016). Ortaöğretim Öğrencilerine Yönelik Bilgi Güvenliği Farkındalık Ölçeği (BGFÖ) Geliştirme Çalışması. İlköğretim Online, 15(2). https://doi.org/10.17051/io.2016.27218.
22. Hadlington, L. (2017). Human factors in cybersecurity; examining the link between Internet addiction, impulsivity, attitudes towards cybersecurity, and risky cybersecurity behaviours. Heliyon, 3(7), e00346. https://doi.org/10.1016/j.heliyon.2017.e00346
23. He, W., & Zhang, Z. J. (2019). Enterprise cybersecurity training and awareness programs: Recommendations for success. Journal of Organizational Computing and Electronic Commerce, 29(4), 249–257. https://doi.org/10.1080/10919392.2019.1611528
24. Hijji, M., & Alam, G. (2022). Cybersecurity Awareness and Training (CAT) Framework for Remote Working Employees. Sensors, 22(22), 8663. https://doi.org/10.3390/s22228663
25. Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104-121. https://doi.org/10.1177/109442819800100106
26. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.
27. Jacobs, C., Horsch, A., Adda, M., & Shiraz, M. (2023). Measuring the Effectiveness of Corporate Security Awareness Programs: A Systematic Review. Computers & Security, 126, 103055. https://doi.org/10.1016/j.cose.2022.103055
28. Kannelønning, K., & Katsikas, S. K. (2023). A systematic literature review of how cybersecurity-related behavior has been assessed. Information & Computer Security, 31(5), 631-660. https://doi.org/10.1108/ics-08-2022-0139
29. Kävrestad, J., Burvall, F., & Nohlberg, M. (2024). A taxonomy of factors that contribute to organizational cybersecurity awareness (CSA). Information & Computer Security. https://doi.org/10.1108/ics-11-2023-0209
30. Kävrestad, J., Nohlberg, M., & Ramezanian, S. (2024). A Taxonomy of Cybersecurity Awareness Activities in Organizations. Computers & Security, 138, 103654. https://doi.org/10.1016/j.cose.2023.103654
31. Keser, H., & Güldüren, C. (2015). Development of information security awareness scale. Kastamonu Education Journal, 23(3), 1167–1184. https://kefdergi.kastamonu.edu.tr/index.php/Kefdergi/article/view/531/233
32. Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs: The Four Levels (3rd ed.). Berrett-Koehler Publishers.
33. Korovessis, P., Furnell, S., Papadaki, M., & Haskell-Dowland, P. (2017). A toolkit approach to information security awareness and education. Journal of Cybersecurity Education, Research and Practice, 2017(2), Article 5.
34. Kovačević, M. (2022). Online Deliberation and Personal Identity. EMERGE 2022 Book of Abstracts, 32-33.
35. Kruger, H. A., & Kearney, W. D. (2006). A prototype for assessing information security awareness. Computers & Security, 25(4), 289-296. https://doi.org/10.1016/j.cose.2006.02.008
36. Kumar, S., Biswas, B., Bhatia, M. S., et al. (2021). Antecedents for enhanced level of cyber-security in organisations. Journal of Enterprise Information Management, 35(4/5), 1094-1119. https://doi.org/10.1108/JEIM-06-2020-0240
37. Langner, R. (2011). Stuxnet: Dissecting a cyberwarfare weapon. IEEE Security & Privacy, 9(3), 49-51. https://doi.org/10.1109/MSP.2011.67
38. Li, L., He, W., Xu, L., Ash, I., Anwar, M., & Yuan, X. (2019). Investigating the impact of cybersecurity policy awareness on employees’ cybersecurity behavior. International Journal of Information Management, 45, 13-24. https://doi.org/10.1016/j.ijinfomgt.2018.10.017
39. MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84-99. https://doi.org/10.1037/1082-989X.4.1.84
40. Marnewick, A., & von Solms, S. H. (2022). Measuring Cybersecurity Awareness: A Comprehensive Approach. Information & Computer Security, 30(3), 437-458. https://doi.org/10.1108/ICS-09-2021-0138
41. Mihai, I. C. (2024). AR-in-a-Box: A Structured 8-Step Framework for Cybersecurity Awareness. Proceedings of CyberCon 2024. https://doi.org/10.19107/cybercon.2024.05
42. Nikel, M., & Amaechi, C. V. (2022). An Empirical Study of Employee Awareness on Cybersecurity in Cameroon. Journal of Cybersecurity and Privacy, 2(2), 374-397. https://doi.org/10.3390/jcp2020020
43. Nwachukwu, U., Vidgren, J., Niemimaa, M., et al. (2023). Do SETA Interventions Change Security Behavior? – A Literature Review. Proceedings of the 56th Hawaii International Conference on System Sciences, 6344-6353. https://doi.org/10.24251/hicss.2023.762
44. Parsons, K., McCormac, A., Butavicius, M., Pattinson, M., & Jerram, C. (2017). Determining employee awareness using the Human Aspects of Information Security Questionnaire (HAIS-Q). Computers & Security, 42, 165-176. https://doi.org/10.1016/j.cose.2013.12.003
45. Pasipamire, E. (2025). Building a Culture of Cybersecurity Awareness in Libraries: A Systematic Review of Best Practices and Frameworks. Proceedings of the European Conference on Information Warfare and Security, 24(1), 357-365. https://doi.org/10.34190/eccws.24.1.3608
46. Puhakainen, P., & Siponen, M. (2010). Improving employees’ compliance through information systems security training: An action research study. MIS Quarterly, 34(4), 757-778. https://doi.org/10.2307/25750704
47. Salahdine, F., & Kaabouch, N. (2019). Social Engineering Attacks: A Survey. Future Internet, 11(4), 89. https://doi.org/10.3390/fi11040089
48. Schütz, A. E., & Fertig, T. (2023). The Forgotten Model – Validating the Integrated Behavioral Model in Context of Information Security Awareness. Proceedings of the 56th Hawaii International Conference on System Sciences, 6809-6818. https://doi.org/10.24251/hicss.2023.828
49. Shaw, R. S., Chen, C. C., Harris, A. L., & Huang, H. J. (2009). The impact of information richness on information security awareness training effectiveness. Computers & Education, 52(1), 92-100. https://doi.org/10.1016/j.compedu.2008.06.011
50. Sulaiman, N. S., Fauzi, M. A., Hussain, S., et al. (2022). Cybersecurity Behavior among Government Employees: The Role of Protection Motivation Theory and Responsibility in Mitigating Cyberattacks. Information, 13(9), 413. https://doi.org/10.3390/info13090413
51. Steinbart, P. J., Raschke, R. L., Gal, G., & Dilla, W. N. (2016). SECURQUAL: An instrument for evaluating the effectiveness of enterprise information security programs. Journal of Information Systems, 30(1), 71–92. https://doi.org/10.2308/isys-51257
52. Tabachnick, B. G., & Fidell, L. S. (2013). Using Multivariate Statistics (6th ed.). Pearson.
53. Vortia, W. (2025). Modelling cybersecurity awareness, perceived threats and secure online behavioral intentions among Ghanaian university students: A PLS-SEM Approach. Magna Scientia Advanced Research and Reviews, 14(2), 114-131. https://doi.org/10.30574/msarr.2025.14.2.0094
54. Vrhovec, S., & Markelj, B. (2024). We need to aim at the top: Factors associated with cybersecurity awareness of cyber and information security decision-makers. arXiv preprint. https://doi.org/10.48550/arxiv.2404.04725
55. Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806-838. https://doi.org/10.1177/0011000006288127
56. Zwilling, M., Klien, G., Lesjak, D., Wiechetek, Ł., Çetin, F., & Başım, H. N. (2022). Cyber security awareness, knowledge and behavior: A comparative study. Journal of Computer Information Systems, 62(1), 82–97.