A Meta-Analytic Reliability Generalization Study of the Oxford Happiness Scale in Turkish Sample

Abstract

The purpose of this study was to analyze the meta-analytical reliability generalization of short form and long form of the Oxford Happiness Scale (OHS) for Turkish sample. In addition, how different moderator variables affect reliability coefficients was examined. A number of criteria have been set to determine the studies to be included in meta-analysis. Of 95 Cronbach’s Alpha coefficients obtained from 92 studies that were selected according to criteria were included in the meta-analysis. In the data analysis, reliability generalization based on meta-analysis was used. The effect of moderator variables on variability in reliability estimations as effect size was examined by Analog ANOVA. As a result of the research, it was found that the mean alpha was .81 for overall studies; .76 for the short form and .87 for the long form of OHS. In addition, it was concluded that number of items had a statistically significant effect on the reliability estimation in terms of heterogeneity of true effect sizes, and sample type had a statistically significant effect on the reliability estimation for OHS (long-form). But sample type had no effect on the reliability estimation for OHS-S (short-form), and field of study had no effect for both short and long form reliability estimates.

Keywords

• Reliability generalization
• meta-analysis
• Oxford happiness scale.

References

1. Aguayo, R., Vargas, C., Emilia, I., & Lozano, L. M. (2011). A meta-analytic reliability generalization study of the Maslach Burnout Inventory. International Journal of Clinical and Health Psychology, 11(2), 343-361. Retrieved from http://www.redalyc.org/articulo.oa?id=33716996009
2. Anastasi, A. (1982) Psychological testing (5th ed.). New York: Macmillan.
3. Argyle, M., Martin, M., & Crossland, J. (1989). Happiness as a function of personality and social encounters. In J. P. Forgas & J. M. Innes (Eds.), Recent advances in social psychology: An international perspective (pp. 189- 203). Amsterdam: North Holland, Elsevier Science.
4. Argyle, M., Martin, M., & Lu, L. (1995). Testing for stress and happiness: The role of social and cognitive factors. In C. D. Spielberger, I. G. Sarason, J. M. T. Brebner, E. Greenglass, P. Laungani, & A. M. O'Roark (Eds.), Series in stress and emotion: Anxiety, anger, and curiosity, (Vol. 15). Stress and emotion: Anxiety, anger, and curiosity (p. 173-187). Washington, DC: Taylor & Francis.
5. Armor, D. J. (1973). Theta reliability and factor scaling. Sociological Methodology, 5(1973-1974), 17-50. doi: 10.2307/270831
6. Barnes, L. L., Harp, D., & Jung, W. S. (2002). Reliability generalization of scores on the Spielberger state-trait anxiety inventory. Educational and Psychological Measurement, 62(4), 603-618. doi: 10.1177/0013164402062004005
7. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50(4), 1088-1101. doi: 10.2307/2533446
8. Beretvas, S. N., Meyers, J. L., & Leite, W. L. (2002). A reliability generalization study of the Marlowe-Crowne Social Desirability Scale. Educational and Psychological Measurement, 62(4), 570-589. doi: 10.1177/0013164402062004003

9. Bonett, D. G. (2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics, 27(4), 335-340. doi: 10.3102/10769986027004335
10. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2013). Introduction to meta-analysis. UK: John Wiley & Sons.
11. Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PloS One, 5(12), 1-10. doi: 10.1371/journal.pone.0014331
12. Capraro, R. M., & Capraro, M. M. (2002). Myers-briggs type indicator score reliability across: Studies a meta-analytic reliability generalization study. Educational and Psychological Measurement,62(4), 590-602. doi: 10.1177/0013164402062004004
13. Caruso, J. C. (2000). Reliability generalization of the NEO personality scales. Educational and Psychological Measurement, 60(2), 236-254. doi: 10.1177/00131640021970484
14. Caruso, J. C., Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J. D. (2001). Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization study. Educational and Psychological Measurement, 61(4), 675-689. doi: 10.1177/00131640121971437
15. Compton, W. C., & Hoffman, E. (2019). Positive psychology: The science of happiness and flourishing. Thousands Oak, CA: SAGE Publications.
16. Crocker, L., & Algina, J. (2008). Introduction to classical and modern test theory. Ohio: Cengage Learning.
17. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. doi: 10.1007/bf02310555
18. Cronbach, L. J., Schönemann, P., & McKie, D. (1965). Alpha coefficients for stratified-parallel tests. Educational and Psychological Measurement, 25(2), 291-312. doi: 10.1177/001316446502500201
19. Demir, Ü. (2020). Aile özellikleri ve mutluluk: Çanakkale’de lise öğrencileri üzerine bir araştırma. Kastamonu Eğitim Dergisi, 28(3), 1296-1306. Retrieved from https://dergipark.org.tr/en/download/article-file/1109602
20. Doğan, T., & Çötok, N. A. (2011). Oxford mutluluk ölçeği kısa formunun Türkçe uyarlaması: Geçerlik ve güvenirlik çalışması. Türk Psikolojik Danışma ve Rehberlik Dergisi, 4(36), 165-172. Retrieved from http://www.turkpdrdergisi.com/index.php/pdr/article/view/100/101
21. Doğan, T., & Sapmaz, F. (2012). Oxford mutluluk ölçeği Türkçe formunun psikometrik özelliklerinin üniversite öğrencilerinde incelenmesi. Düşünen Adam Psikiyatri ve Nörolojik Bilimler Dergisi, 25(4), 297-304. Retrieved from https://dusunenadamdergisi.org/storage/upload/pdfs/1586247664-tr.pdf
22. Duval, S., & Tweedie, R. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95(449), 89-98. doi: 10.1080/01621459.2000.10473905
23. Duval, S., & Tweedie, R. (2000b). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455-463. doi: 10.1111/j.0006-341x.2000.00455.x
24. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. Bmj, 315(7109), 629-634. doi: 10.1136/bmj.315.7109.629 Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), The American Council on Education/Macmillan series on higher education. Educational measurement (pp. 105-146). New York: Macmillan Publishing Co., Inc.; American Council on Education.
25. Field, A. P. (2003). The problems in using fixed-effects models of meta-analysis on real-world data. Understanding Statistics: Statistical Issues in Psychology, Education, and the Social Sciences, 2(2), 105-124. doi: 10.1207/S15328031US0202_02
26. Francis, L. J., & Crea, G. (2018). Happiness matters: Exploring the linkages between personality, personal happiness, and work-related psychological health among priests and sisters in Italy. Pastoral Psychology, 67(1), 17-32. doi: 10.1007/s11089-017-0791-z
27. Francis, L. J., & Katz, Y. J. (2000). Internal consistency reliability and validity of the Hebrew translation of the Oxford Happiness Inventory. Psychological Reports, 87(1), 193-196. doi: 10.2466/pr0.2000.87.1.193
28. Francis, L. J., Brown, L. B., Lester, D., & Philipchalk, R. (1998). Happiness as stable extraversion: A cross-cultural examination of the reliability and validity of the Oxford Happiness Inventory among students in the UK, USA, Australia, and Canada. Personality and Individual Differences, 24(2), 167-171. doi: 10.1016/S0191-8869(97)00170-0
29. Francis, L. J., Ok, Ü., & Robbins, M. (2017). Religion and happiness: A study among university students in Turkey. Journal of religion and health, 56(4), 1335-1347. Retrieved from http://glyndwr.repositorytest.guildhe.ac.uk:8080/9182/1/Robbins_Religion_and_happiness.pdf
30. Gilmer, J. S., & Feldt, L. S. (1983). Reliability estimation for a test with parts of unknown lengths. Psychometrika, 48(1), 99-111. doi: 10.1007/BF02314679
31. Graham, J. M., Diebels, K. J., & Barnow, Z. B. (2011). The reliability of relationship satisfaction: A reliability generalization meta-analysis. Journal of Family Psychology, 25(1), 39-48. retrieved from https://pdfs.semanticscholar.org/4bad/75bc14e345e7dabfc01888b50b5ef502305d.pdf
32. Graham, J. M., Liu, Y. J., & Jeziorski, J. L. (2006). The dyadic adjustment scale: A reliability generalization meta‐analysis. Journal of Marriage and Family, 68(3), 701-717. doi:10.1111/j.1741-3737.2006.00284.x
33. Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255-282. doi: 10.1007/bf02288892
34. Hakstian, A. R., & Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41(2), 219-231. doi: 10.1007/BF02291840
35. Hanson, W. E., Curry, K. T., & Bandalos, D. L. (2002). Reliability generalization of working alliance inventory scale scores. Educational and Psychological Measurement, 62(4), 659-673. doi: 10.1177/0013164402062004008
36. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89. doi: 10.1080/19312450709336664
37. Heise, D. R., & Bohrnstedt, G. W. (1970). Validity, invalidity, and reliability. Sociological Methodology, 2(1970), 104-129. doi: 10.2307/270785
38. Henson, R. K., Kogan, L. R., & Vacha-Haase, T. (2001). A reliability generalization study of the teacher efficacy scale and related instruments. Educational and Psychological Measurement, 61(3), 404-420. doi: 10.1177/00131640121971284
39. Hess, T. J., McNab, A. L., & Basoglu, K. A. (2014). Reliability generalization of perceived ease of use, perceived usefulness, and behavioral intentions. Mis Quarterly, 38(1), 1-28. Retrieved from https://www.jstor.org/stable/26554866
40. Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta‐analysis. Statistics in Medicine, 21(11), 1539-1558. doi: 10.1002/sim.1186
41. Hills, P., & Argyle, M. (2002). The Oxford Happiness Questionnaire: A compact scale for the measurement of psychological well-being. Personality and Individual Differences, 33(7), 1073-1082. doi: 10.1016/S0191-8869(01)00213-6
42. Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychological Methods, 11(2), 193-206. doi: 10.1037/1082-989X.11.2.193
43. İlhan, M., & Güler, N. (2017). Likert tipi ölçeklerde olumsuz madde ve kategori sayısı sorunu: Rasch modeli ile bir inceleme. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 8(3), 321-343. doi: 10.21031/epod.321057
44. Kline, T. J. (2005). Psychological testing: A practical approach to design and evaluation. Thousand Oaks, CA: Sage Publications.
45. Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: Sage.
46. Kristof, W. (1963). The statistical theory of stepped-up reliability coefficients when a test has been divided into several equivalent parts. Psychometrika, 28(3), 221-238. doi: 10.1007/bf02289571
47. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2(3), 151-160. doi: 10.1007/bf02288391
48. Li, A., & Bagger, J. (2007). The Balanced inventory of desirable responding (BIDR): A reliability generalization study. Educational and psychological measurement, 67(3), 525-544. doi: 10.1177/0013164406292087
49. Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology, 62(10), 1-34. doi: 10.1016/j.jclinepi.2009.06.006
50. Lin, C. Y., Imani, V., Griffiths, M. D., & Pakpour, A. H. (2020). Psychometric properties of the Persian Generalized Trust Scale: Confirmatory factor analysis and Rasch models and relationship with quality of life, happiness, and depression. International Journal of Mental Health and Addiction. doi: 10.1007/s11469-020-00278-0
51. Lu, L., & Shih, J. B. (1997). Personality and happiness: Is mental health a mediator? Personality and Individual Differences, 22(2), 249-256. doi: 10.1016/S0191-8869(96)00187-0
52. McDonald, R. P. (1985). Factor analysis and related methods. London: Lawrence Erlbaum Associates Publishers.
53. McDonald, R. P. (1999). Test theory: A unified treatment. London: Lawrence Erlbaum Associates Publishers.
54. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009) Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med, 6(7), 1-6. doi: 10.1371/journal.pmed.1000097
55. Nilsson, J. E., Schmidt, C. K., & Meek, W. D. (2002). Reliability generalization: An examination of the career decision-making self-efficacy scale. Educational and Psychological Measurement, 62(4), 647-658. doi: 10.1177/0013164402062004007
56. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
57. Okur, S., & Totan, T. (2019). Psikolojik iyi oluşu değerlendiren Bradburn duygulanım dengesi ölçeğinin Türkçede incelenmesi. Adnan Menderes Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 6(2), 1-12. Retrieved from https://dergipark.org.tr/en/download/article-file/935147
58. Raju, N. S. (1977). A generalization of coefficient alpha. Psychometrika, 42(4), 549-565. doi: 10.1007/bf02295978
59. Revelle, W. (1979). Hierarchical cluster analysis and the internal structure of tests. Multivariate Behavioral Research, 14(1), 57-74. doi: 10.1207/s15327906mbr1401_4
60. Romano, J. L., Kromrey, J. D., & Hibbard, S. T. (2010). A Monte Carlo study of eight confidence interval methods for coefficient alpha. Educational and Psychological Measurement, 70(3), 376-393. doi: 10.1177/0013164409355690
61. Sánchez‐Meca, J., López‐López, J. A., & López‐Pina, J. A. (2013). Some recommended statistical analytic practices when reliability generalization studies are conducted. British Journal of Mathematical and Statistical Psychology, 66(3), 402-425. doi: 10.1111/j.2044-8317.2012.02057.x
62. Shields, A. L., & Caruso, J. C. (2003). Reliability generalization of the alcohol use disorders identification test. Educational and Psychological Measurement, 63(3), 404-413. doi: 10.1177/0013164403063003004
63. Shields, A. L., & Caruso, J. C. (2004). A reliability induction and reliability generalization study of the CAGE questionnaire. Educational and Psychological Measurement, 64(2), 254-270. doi: 10.1177/0013164403261814
64. Taşdibi-Ünlü, F. (2019). Üniversite öğrencilerinde yaşam değeri, yaşamın anlamı ve sosyal iyi olmanın mutluluğu yordamadaki rolü (Yüksek lisans tezi). Muğla Üniversitesi Eğitim Bilimleri Enstitüsü, Muğla. http://tez2.yok.gov.tr/
65. Thompson, B., & Cook, C. (2002). Stability of the reliability of libqual+™ scores a reliability generalization meta-analysis study. Educational and Psychological Measurement, 62(4), 735-743. doi: 10.1177/0013164402062004013
66. Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60(2), 174-195. doi: 10.1177/0013164400602002
67. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58(1), 6-20. doi: 10.1177/0013164498058001002
68. Vacha-Haase, T., & Thompson, B. (2011). Score reliability: A retrospective look back at 12 years of reliability generalization studies. Measurement and Evaluation in Counseling and Development, 44(3), 159-168. doi: 10.1177/0748175611409845
69. Vacha-Haase, T., Henson, R. K., & Caruso, J. C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62(4), 562-569. doi: 10.1177/0013164402062004002
70. Vicent, M., Rubio-Aparicio, M., Sánchez-Meca, J., & Gonzálvez, C. (2019). A reliability generalization meta-analysis of the child and adolescent perfectionism scale. Journal of Affective Disorders, 245(2019), 533-544. doi: 10.1016/j.jad.2018.11.049
71. Viswesvaran, C., & Ones, D. S. (2000). Measurement error in “Big Five Factors” personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60(2), 224-235. doi: 10.1177/00131640021970475
72. Wallace, K. A., & Wheeler, A. J. (2002). Reliability generalization of the life satisfaction index. Educational and Psychological Measurement, 62(4), 674-684. doi: 10.1177/0013164402062004009
73. Wheeler, D. L., Vassar, M., Worley, J. A., & Barnes, L. L. (2011). A reliability generalization meta-analysis of coefficient alpha for the Maslach Burnout Inventory. Educational and Psychological Measurement, 71(1), 231-244. doi: 10.1177/0013164410391579
74. Yıldırım, O., & Sezer, Ö. (2020). The relationship between nomophobia and trait anxiety, basic psychological needs, happiness in adolescents. Journal of Human Sciences, 17(2), 535-547. Retrieved from https://j-humansciences.com/ojs/index.php/IJHS/article/view/5917/3375
75. Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60(2), 201-223. doi: 10.1177/00131640021970466