Meta-Analytic Reliability Generalization Study of Perceived Stress Scale In Türkiye Sample

Abstract

The aim of the study is to examine the meta-analytic reliability generalization of the 14, 10, and 8-item forms of the Perceived Stress Scale, developed by Cohen et al. in 1983, for theses produced in Turkey. For this purpose, how different moderator variables affect the reliability coefficients and publication bias were also examined 81 Cronbach Alpha coefficients from 78 studies selected in accordance with the established criteria were included in the meta-analysis. Reliability generalization was used in the analysis of the data. Analyzes were made with the random effects model using the coefficient values converted from the Bonett method. In the study, the effect size value was found to be .82 (95% CI: .80, .83), and it was concluded that the sample type (α= .81) and the study area (α= .81) moderator variables had a statistically significant effect on the reliability estimation in terms of heterogeneity of effect sizes. This revealed that these two moderator variables affected the estimations of the reliability coefficients. In addition, it has been observed that other moderator variables such as age, gender, and the number of items in the scale are not sources of heterogeneity and have no effect on reliability estimation. From this, it was deduced that the scale works well enough to generalize to different situations with different item numbers. Finally, according to the results of the analysis for the determination of publication bias, it was seen that there was no publication bias in the study.

Keywords

• Reliability generalization
• Meta-analysis
• Perceived stress scale
• Cronbach alpha coefficient
• Bonett method

References

1. Acar, O. (2024). The relationship between emotional eating, perceived stress and perceived social support in people with and without a diagnosis of bipolar disorder. [Unpublished Master Thesis]. İstanbul Arel University.
2. Agbo, A. A. (2010). Cronbach's alpha: review of limitations and associated recommendations, Journal of Psychology in Africa, 20(2), 233-239. http://dx.doi.org/10.1080/14330237.2010.10820371
3. Aguayo, R., Vargas, C., & Lozano, L. M., (2011). A meta-analytic reliability generalization study of the maslach burnout inventory. International Journal of Clinical and Health Psychology, 11(2), 343-361.
4. Alenezi, S. S. (2023). The reliability of Maslach burnout inventory in Arab studies: Reliability generalization meta-analytic study. International Journal for Research in Education, 47(5),46-75. http://doi.org/10.36771/ijre.47.5.23-pp46-75
5. Allan, T. A. (2021). Resiliency: a systematic review of adult characterological measures of resilience and reliability and validity generalization studies of the brief resilience scale. [Unpublished Doctoral Dissertations]. University of Ottawa Social Sciences.
6. Alzahrani, D. T. (2016). A reliability generalization study of the brief symptom inventory-18. [Unpublished Master Thesis]. University of Denver.
7. Andreou, E., Alexopoulos, E. C., Lionis, C., Varvogli, L., Gnardellis, C., Chrousos, G. P. & Darviri, C. (2011). Perceived stress scale: reliability and validity study in greece. International Journal of Environmental Research and Public Health, 8(8):3287-3298. https://doi.org/10.3390/ijerph8083287
8. Ataman Temizel, E. & Dağ, İ. (2014). Relationships between stressful life events, cognitive emotion regulation strategies, depressive symptoms and anxiety level. Journal of Clinical Psychiatry, 17(1).

9. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50(4), 1088-1101. doi: 10.2307/2533446
10. Beretvas, S. N., Meyers, J. L., & Leite,W. L. (2002). A reliability generalization study of the Marlowe-Crowne Social Desirability Scale. Educational and Psychological Measurement, 62(4), 570–589.
11. Beretvas, S. N., Suizzo, M., Durham, J. & Yarnell, L. M. (2008). A reliability generalization study of scores on rotter’s and nowicki-strickland’s locus of control scales. Educational and Psychological Measurement, 68(1), 97-119.
12. Bilge A., Öğce, F., Genç, R. E., & Oran, N. T. (2009). Psychometric relevance of the Turkish version of the Perceived Stress Scale (PSS). Ege University School of Nursing Journal, 2(25), 61-72
13. Birol, L., & Akdemir, N. (2003). Internal Medicine and Nursing Care 1. Edition, Vehbi Koç Foundation Publications.
14. Bonett, D. G. (2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics 27(4): 335-340.
15. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2013). Introduction to meta-analysis. UK: John Wiley & Sons.
16. Boyd, D., & Bee, H. (2014). Lifespan Development (7th ed.). New York, NY: Pearson International.
17. Büyüköztürk, Ş., Kılıç Çakmak, E., Akgün, Ö. E., Karadeniz, Ş. & Demirel, F. (2019). Scientific research methods in education. Pegem Academy.
18. Camci, G. B. (2021). Determination of the relationship between job stress and burnout levels of nurses and their occupational and life satisfaction levels. [Unpublished Master Thesis]. Atatürk University.
19. Cohen, J. (1960), A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20(1), 37-46
20. Cohen S. (1994), Perceived stress scale, Retrieved from https://www.mindgarden.com/documents/PerceivedStressScale.pdf on 18.10.23.
21. Cohen, S., Kamarck, T. & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24(1), 386-396.
22. Cohen, S., Kessler, R.C. & Gordon, L.U. (1997). Strategies for measuring stress in studies of psychiatric and physical disorders. S. Cohen, R.C. Kessler & L.U. Gordon (Ed.), Measuring stress: A guide for health and social scientists 122–148. New York: Oxford University Press.
23. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart, & Winston.
24. Cronbach, L. J. (1947). Test "reliability": its meaning and determination. Psychometrica 12, 1-16.
25. Çalışkan Altıntaş, D. (2024). The relationship between marital adjustment and perceived prenatal stress in pregnant women, [Unpublished Master Thesis], Tokat Gaziosmanpaşa University.
26. Çelik-Örücü, M., & Demir, A. (2009). Psychometric evaluation of perceived stressscale for Turkish university students. Stress and Health, 25(1), 103-109.
27. Duval, S., & Tweedie, R. (2000b). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455-463. doi: 10.1111/j.0006-341x.2000.00455.x
28. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. Bmj, 315(7109), 629-634. doi: 10.1136/bmj.315.7109.629
29. Erci B. (2006). Reliability and validity of the Turkish version of perceived stress scale. A.Ü. Journal of the School of Nursing, 9(1):58-64.
30. Eser, M. T. & Aksu, G. (2021). Beck depression inventory: a study for meta-analytical reliability generalization. Pegem Journal of Education and Instruction, 11(3),88-101
31. Eskin, M., Harlak H., Demirkıran, F. & Dereboy, Ç. (2013). Adaptation of the perceived stress scale to Turkish: reliability and validity analysis. New Symposium Journal 51 (3): 132‐140.
32. Esparza- Reig, J., Guillen-Riquelme, A., Marti-Vilar, M. & Gonzalez-Sala, F. (2021). A reliability generalization meta-analysis of the south oaks gambling screen (sogs). Psicothema 33(3). 490-499.
33. Field, A. & Gillett, R. (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63(3), 665-694.
34. Gallagher, M. W., Zvolensky, M. J., Long, L. J., Rogers, A. H. & Garey, L. (2020). The impact of covid-19 experiences and associated stress on anxiety, depression, and functional impairment in American adults. Cognitive Therapy and Research, 44(6). https://doi.org/ 10.1007/s10608-020-10143-y.
35. Graves, B.S., Hall. M.E., Dias-Karch, C., Haischer, M.H. & Apter, C. (2021). Gender differences in perceived stress and coping among college students. Plos One. 16(8): e0255634. https://doi.org/10.1371/journal.pone.0255634
36. Haertel, E. (2006). Reliability. Brennan, R. L. (2006). Educational measurement (65-110). 4th ed. Westport (Conn.): Praeger Publishers.
37. Havare, G. (2019). The effect of perceived stress level on job satisfaction: a study on nurses working in a public hospital. [Unpublished Master Thesis]. Bahçeşehir University.
38. Henson, R. K., & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting “reliability generalization” studies. Measurement and Evaluation in Counseling and Development, 35(2), 113–127
39. Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta‐analysis. Statistics in Medicine, 21(11), 1539-1558. doi: 10.1002/sim.1186
40. Hongyan, L., Jianping, X., Jiyue, C. & Yexin, F. (2015). A reliability meta-analysis for 44 items big five inventory: based on the reliability generalization methodology. Advances in Psychological Science. 23(5), 755-765.
41. Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychological Methods, 11(2), 193-206. doi: 10.1037/1082-989X.11.2.193
42. Hussey, I., Alsalti, T., Bosco, F., Elson, M., & Arslan, R. C. (2023). An aberrant abundance of Cronbach’s alpha values at .70. Retrived October 05, 2024, from https://osf.io/preprints/psyarxiv/dm8xn, https://doi.org/10.31234/osf.io/dm8xn
43. Katsarou, A. L., Triposkiadis, F. & Panagiotakos, D. (2013). Perceived stress and vascular disease: where are we now? Angiology, 64(7), 529-534. doi:10.1177/0003319712458963 Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33(1), 159-174.
44. Lazarus, R. S. (1990). Theory-based stress measurement. Psychological Inquiry, 1(1), 3–13. https://doi.org/10.1207/s15327965pli0101_1
45. Lazarus, R.S. (1993). From psychological stress to the emotions: a history of changing outlooks. Annual Reviews Psychology, 44, 1- 21.
46. Lazarus, R. S. & Launier, R. (1978). Stress-related transactions between person and environment. Perspectives in Interactional Psychology, 287–327. doi:10.1007/978-1-4613-3997-7_12
47. Lee, J., Joo, E. & Choi, K. (2012). Perceived stress scale and self-esteem mediate the effects of work-related stress on depression. Stress & Health 29(1) 75-81. https://doi.org/10.1002/smi.2428
48. Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology, 62(10), 1-34. doi: 10.1016/j.jclinepi.2009.06.006
49. Lipsey, M. W. & Wilson, D. B. (2001). Practical meta-analysis. SAGE publications, Inc.
50. Nicolas, N. L., Aparicio, M. R., Ibanez, C. L.& Meca, J. S. (2021). A reliability generalization meta-analysis of the dimensional obsessive-compulsive scale. Psicothema, 33(3),481-489.
51. Orwin, R.G. (1983). A fail-safe N for effecet size in meta-analysis. Journal of Educational Statistics, 8(2), 157-159.
52. Osmanovic-Thunström, A., Mossello, E., Akerstedt, T., Fratiglioni, L. & Wang, H. (2015). Do levels of perceived stress increase with increasing age after age 65? A population-based study, Age and Ageing, 44(5), 828–834. https://doi.org/10.1093/ageing/afv078
53. Özdemir, V., Yıldırım, Y., & Tan, Ş. (2020). A meta-analytic reliability generalization study of the Oxford Happiness Scale in Turkish sample. Journal of Measurement and Evaluation in Education and Psychology, 11(4), 374-404. doi: 10.21031/epod.766266.
54. Özkan, S., Sakal, F. N., Avcı, E. Civil, E. F. & Tunca, M. Z. (2013). Preference of women's birth method and related factors. Turkish Journal of Public Health, 11(2), 59–71.
55. Pedrozo-Pupo, J. C., Pedrozo-Cortes, M. J., Campo-Arias, A. (2020). Perceived stress associated with covid-19 epidemic in Colombia: an online survey. Cad Saúde Pública, 36 (5), https://doi.org/10.1590/0102-311x00090520.
56. Qiu, J., Shen, B., Zhao, M., Wang, Z., Xie, B. & Xu, Y. (2020). A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: implications and policy recommendations. General Psychiatry, 33(2), https://doi.org/ 10.1136/gpsych-2020-100213.
57. Ramírez, M. T. G., & Hernández, R. L. (2007). Factor structure of the perceived stress scale (PSS) in a sample from Mexico. The Spanish Journal of Psychology, 10(1), 199–206. doi:10.1017/S1138741600006466
58. Razurel, C., Kaiser, B., Dupuis, M., Antonietti, J., Citherlet, C. Epiney, M. & Sellenet, C. (2014). Validation of the antenatal perceived stress inventory. Journal of Health Psychology, 19(4), 471-481. doi:10.1177/1359105312473785
59. Rouse, S. V. (2007). Using Reliability Generalization methods to explore measurement error: An illustration using the MMPI–2 PSY–5 scales. Journal of Personality Assessment, 88, 264–275.
60. Santrock, J. W. (2012). Life-span developmental Psychology. New York: McGraw Hill Companies, Inc.
61. Sheu, S., Lin, H. S., & Hwang, S. L. (2002). Perceived stress and physio-psycho-social status of nursing students during their initial period of clinical practice: the effect of coping behaviors. International journal of nursing studies, 39(2), 165–175. https://doi.org/10.1016/s0020-7489(01)00016-5
62. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). p-curve and effect size: correcting for publication bias using only significant results. Perspectives on psychological science: a journal of the Association for Psychological Science, 9(6), 666–681. https://doi.org/10.1177/1745691614553988
63. Singh, S., Pandey, N. M., Datta, M. & Batra, S. (2021). Stress, internet use, substance use and coping among adolescents, young-adults and middle-age adults amid the ‘new normal’ pandemic era, Clinical Epidemiology and Global Health, 12 https://doi.org/10.1016/j.cegh.2021.100885
64. Snoeren, F., Hoefnagels, C. Measuring perceived social support and perceived stress among primary school children in the netherlands. Child Indicators Research 7, 473–486. https://doi.org/10.1007/s12187-013-9200-z
65. Streiner, D.L., Norman, G.R. & Cairney, J. (2015). Health measurement scales: A practical guide to their development and use (5th ed.). Oxford University Press.
66. Şen, S. (2018). Meta-analysis. https://sedatsen.files.wordpress.com/2018/06/meta-analiz.pdf
67. Şen, S. (2021a). A reliability generalization meta-analysis of Runco ideational behavior scale, Creativity Research Journal, 34:2, 178-194, DOI:10.1080/10400419.2021.1960719
68. Şen, S. (2021b). Current developments in meta-analysis. https://drive.google.com/file/d/1FhCXWjNgmkUSBxi_kZWjJXzAh1v7gfZq/view
69. Şen, S. & Yıldırım, İ. (2020). Meta-analysis applications with CMA. Anı Publishing.
70. Şeten, C. (2012). Meta-analysis: an application of the reliability generalization of the multidimensional student life satisfaction scale (mslss). [Unpublished Master Thesis], Akdeniz University.
71. Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174–195.
72. Toprak, F. (2023). The mediating role of psychological flexibility in the relationship between obsessive-compulsive symptoms and shame, guilt and perceived stress. [Unpublished Master Thesis] Yakın Doğu University.
73. Urbina, S. (2004). Essentials of psychological testing. Hoboken, NJ: John Wiley & Sons, Inc.
74. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
75. Vacha-Haase, T., Kogan, L.R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those of test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522. https://doi.org/10.1177/ 00131640021970682
76. Vassar, M. & Bradley, G. (2010) A reliability generalization study of coefficient alpha for the life orientation test, Journal of Personality Assessment, 92(4), 362-370, DOI: 10.1080/00223891.2010.482016
77. Vassar, M. & Crosby, J. W. (2008). A reliability generalization study of coefficient alpha for the ucla loneliness scale. Journal of Personality Assessment. 90(6), 601-607, DOI: 10.1080/00223890802388624
78. Wheeler, D. L., Vassar, M., Worley, J. A., & Barnes, L. L. B. (2011). A Reliability Generalization Meta-Analysis of Coefficient Alpha for the Maslach Burnout Inventory. Educational and Psychological Measurement, 71(1), 231-244. https://doi.org/10.1177/0013164410391579
79. Wilkinson, L., & the APA Task Force on Statistical Inference (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. Yerlikaya, E. E. & İnanç, B. (2007). Psychometric properties of the Turkish translation of the perceived stress scale. IX. National Psychological Counseling and Guidance Congress, 17-19 October, İzmir. Yüksel, Ş. (2024). Factors affecting prenatal perceived stress and prenatal attachment in pregnant women. [Medical Specialty Thesis]. University of Health Sciences.