INVESTIGATING THE RELIABILITY OF CHATGPT IN ASSESSING JOB SATISFACTION

Year 2023, Volume: 5 Issue: 2, 110 - 133, 29.10.2023

Burak Çavuş , Onur Kazancı , Murat Esen

Abstract

Today, there is increasing controversy surrounding the use of artificial intelligence software and programs. Although many organizations are attempting to boost their performance by utilizing the potential of artificial intelligence, controversy about the technology's reliability prevails. This study investigates the reliability of ChatGPT in measuring job satisfaction. The Minnesota Job Satisfaction Questionnaire (MSQ), a reliable scale, and ChatGPT results were compared using the Pearson correlation coefficient Correlation technique. The SPSS 25 analysis findings revealed a strong connection between the two questionnaires. This proves that ChatGPT is a reliable tool for assessing job satisfaction. The ability of the questionnaire to correctly predict the MSQ results provided more evidence of the questionnaire validity. However, the study was limited in its sampling in terms of variety. Additionally, the sample size of the study was relatively small and restricted to almost one organization. The results of this study point to ChatGPT as a potential technology for gaging employee engagement and job satisfaction in businesses. These findings may have consequences for further research on these topics.

Keywords

ChatGPT, Job Satisfaction, MSQ, Organizational Behaviour, Artificial Intelligence

References

• Aamodt, S. P. (2014). Industrial/organizational psychology: An applied approach (6th ed.). Wadsworth, Cengage Learning.
• Armengol-Estap'e, J., Bonet, O.D., & Melero, M. (2021). On the Multilingual Capabilities of Very Large-Scale English Language Models. ArXiv, abs/2108.13349.
• Babbie, E. R. (2016). The basics of social research (7th ed.). Cengage Learning.
• Bakker, A. B., & Demerouti, E. (2017). Job demands-resources theory: Taking stock and looking forward. Journal of Occupational Health Psychology, 22(3), 273-285. ChatGPT (2020). ChatGPT questionnaire, https://www.chatgpt.com/questionnaire (Accessed: 12nd February 2023)
• Chatman, J. A., & Barsade, S. G. (1995). The influence of organizational culture on employee satisfaction and motivation. Journal of Applied Psychology, 80(3), 330-339.
• Cheng, K., Li, Z., Li, C., Xie, R., Guo, Q., He, Y., & Wu, H. (2023). The potential of GPT-4 as an AI-powered virtual assistant for surgeons specialized in joint arthroplasty. Annals of Biomedical Engineering, 51, 1366-1370. https://doi.org/10.1007/s10439-023-03207-z
• Clark, A. E., & Senik, C. (2016). Who compares to whom? The anatomy of income comparisons in Europe. Economic Journal, 126(598), 459-488.
• Demerouti, E., Bakker, A. B., & Schaufeli, W. B. (2005). Burnout and work engagement: A two-sample test of the job demands-resources model. Journal of Applied Psychology, 90(2), 138-148.
• Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE Publications.
• Hackman, J. R., & Oldham, G. R. (1975a). Development of the job diagnostic survey. Journal of Applied Psychology, 60(2), 159-170.
• Hackman, J. R., & Oldham, G. R. (1976b). Motivation through the design of work: Test of a theory. Organizational Behavior and Human Performance, 16(2), 250-279.
• Hakanen, J. J., & Schaufeli, W. B. (2012). Do burnout and work engagement predict depressive symptoms and life satisfaction? A three-wave seven-year prospective study. Journal of Affective Disorders, 141(2-3), 415-424.
• Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business-unit-level relationship between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87(2), 268-279. https://doi.org/10.1037/0021-9010.87.2.268
• Hernandez, R. (2020). ChatGPT: A revolutionary technology for automating customer service, https://www.entrepreneur.com/article/357082 (Accessed: 1st March 2023).
• Hilger, N., & Beauducel, A. (2017). Parallel-forms reliability. In V. Zeigler-Hill, & T. Shackelford (Eds.), Encyclopedia of personality and individual differences (pp. 1337-1345). Springer.
• Ironson, G. H., Smith, P. C., Brannick, M. T., Gibson, W. M., & Paul, K. B. (1989). Construction of a job in general scale: A comparison of global, composite, and specific measures. Journal of Applied Psychology, 74(2), 193-200.
• Judge, T. A., Bono, J. E., Thoresen, C. J., & Patton, G. K. (2001). The job satisfaction-job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127(3), 376-407.
• Kunin, T. (1955). The construction of a new type of attitude measure. Personnel Psychology, 8(1), 65-77.
• Locke, E. A. (1976). The nature and causes of job satisfaction. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 1297-1349). Chicago, IL: Rand McNally.
• Luthans, F., & Youssef-Morgan, C. M. (2017). Psychological capital: An evidence-based positive approach. Annual Review of Organizational Psychology and Organizational Behavior, 4, 339-366.
• Mayo, E. (1933). The human problems of an industrial civilization. Macmillan.
• McCarthy, J., Minsky, M. L., Rochester, N., & Shannon, C. E. (1955). A proposal for the Dartmouth summer research project on artificial intelligence. AI Magazine, 27(4), 12-14.
• Miotto, Marilù & Rossberg, Nicola & Kleinberg, Bennett. (2022). Who is GPT-3? An exploration of personality, values and demographics. arXiv preprint arXiv:2209.14338. https://doi.org/10.48550/arXiv.2209.14338
• Newell, A., & Simon, H. A. (1961). GPS, A program that simulates human thought. In E. A. Feigenbaum, & J. Feldman (Eds.), Computers and thought (pp. 279-293). McGraw.
• Nguyen, C. M. A., & Ha, M.-T. (2023). The interplay between internal communication, employee engagement, job satisfaction, and employee loyalty in higher education institutions in Vietnam. Humanities and Social Sciences Communications, 10, 329. https://doi.org/10.1057/s41599-023-01806-8
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
• OpenAI. (2023). GPT-3: The third generation of OpenAI’s generative pre-trained transformer, https://openai.com/blog/gpt-3/ (Accessed: 9th February 2023).
• OpenAI. (2023). OpenAI and Microsoft extend partnership, https://openai.com/blog/openai-and-microsoft-extend-partnership/ (Accessed: 28th April 2023).
• Ostroff, C. (1992). The relationship between satisfaction, attitudes, and performance: An organizational level analysis. Journal of Applied Psychology, 77(6), 963-974.
• Oswald, A. J., Proto, E., & Sgroi, D. (2015). Happiness and productivity. Journal of Labor Economics, 33(4), 789-822.
• Pereira, F., & Schabes, Y. (1992). Inside-outside reestimation from partially bracketed corpora. Proceedings of the 30th annual meeting on Association for Computational Linguistics (pp. 128-135). Association for Computational Linguistics.
• Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257-286.
• Dale, R. (2021). Gpt-3: What’s it good for?. Natural Language Engineering, 27(1), 113-118.
• Rosen, C. C., Levy, P. E., & Hall, R. J. (2016). Placing perceptions of politics in the context of the feedback environment, employee attitudes, and job performance. Journal of Applied Psychology, 101(10), 1435-1448.
• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536.
• Saks, A. M. (2019). Antecedents and consequences of employee engagement revisited. Journal of Organizational Effectiveness: People and Performance, 6(1), 19-38.
• Smith, P. C., Kendall, L. M., & Hulin, C. L. (1969). The measurement of satisfaction in work and retirement: A strategy for the study of attitudes. Rand McNally.
• Spector, P. E. (1985). Measurement of human service staff satisfaction: Development of the Job Satisfaction Questionnaire. American Journal of Community Psychology, 13(5), 693-713.
• Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient Alpha and internal consistency. Journal of Personality Assessment, 80(1), 99-103.
• Sulea, C., Haţegan, I. G., Maricutoiu, L. P., & Schaufeli, W. B. (2015). Work engagement as mediator between job characteristics and positive and negative extra-role behaviors. Career Development International, 20(6), 590-609.
• Taylor, F. W. (1911). Principles of scientific management. Harper & Brothers.
• Tims, M., Bakker, A. B., & Derks, D. (2013). The impact of job crafting on job demands, job resources, and well-being. Journal of Occupational Health Psychology, 18(2), 230-240. Weber, M. (1947). The theory of social and economic organization. Free Press.
• Weiss, D. J., Dawis, R. V., England, G. W., & Lofquist, L. H. (1967). Manual for the minnesota satisfaction questionnaire. Minnesota Studies in Vocational Rehabilitation, XXII. University of Minnesota.