The Replication Crisis and Psychotherapy Efficacy Studies

Abstract

In the last ten years, the replication crisis, which has manifested itself and shaken the field of psychology profoundly, has led researchers in every subfield of psychology to reconsider research findings and general assumptions. Though late to this trend, clinical psychology researchers have started to participate in these discussions. Psychotherapy efficacy studies, carried out over many years, constitute a major experimental area of clinical psychology. Psychotherapy efficacy studies are conducted to evaluate the effects of psychological treatments on various mental disorders. The findings of the replication crisis indicate that randomized controlled trials, which are frequently used to examine the efficacy of psychotherapies and meta-analyses that help evaluate the results of randomized controlled trials, have several methodological problems and biases. Randomized controlled trials have problems and biases such as low sample size, allegiance bias, selective outcome reporting, strictness of inclusion or exclusion criteria, using wait-list as a control group, problems of randomization and blinding, exclusion of drop-out subjects from the analyses, and ignoring psychotherapists effects. The main methodological issues of meta-analyses are related to publication bias and the inclusion of low-quality studies in meta-analyses. Research over the years has demonstrated that psychotherapies are effective tools for treating various mental disorders. However, there are two critical points to note. First, the studies showing the efficacy of psychotherapies barely meet current scientific methodological standards. Second, the real-life effectiveness of psychotherapies may be lower than those found in clinical trials. This review article aims to increase awareness regarding replication crisis within the field of clinical psychology and strengthen the methodological approach of future studies by addressing the methodological problems of and suggesting solutions for psychotherapy efficacy studies/meta-analyses

Keywords

• Psychotherapy
• replication crisis
• randomized controlled trail
• meta-analysis
• bias
• dodo bird verdict

Tekrarlama Krizi ve Psikoterapi Etkililik Çalışmaları

Abstract

Son 10 yıldır psikoloji alanında kendini gösteren ve alanı derinden sarsan tekrarlama krizi, psikolojinin her alt alanını, araştırma bulgularını ve genel kabullerini yeniden değerlendirmesi için harekete geçirmiştir. Klinik psikoloji alanının geç de olsa tekrarlama krizi tartışmalarına katılmaya başladığı görülmektedir. Uzun yıllardır gerçekleştirilen psikoterapi etkililik (efficacy) çalışmaları, klinik psikoloji içindeki başlıca deneysel alanlardan birini oluşturmaktadır. Psikoterapi etkililik çalışmaları çeşitli psikolojik tedavilerin psikolojik rahatsızlıklar üzerindeki etkilerini değerlendirmek için gerçekleştirilmektedir. Tekrarlama krizi bağlamında ortaya çıkan bulgular, psikoterapilerin etkililiklerini değerlendirmek için sıkça kullanılan seçkisiz kontrollü çalışmaların ve bu çalışmaların sonuçlarının toplu olarak değerlendirilmesine yardımcı olan meta-analizlerin pek çok yöntemsel problem ve yanlılık içerdiğini ortaya çıkarmıştır. Seçkisiz kontrollü çalışmalar düşük örneklem sayısı, bağlılık yanlılığı, seçici sonuç raporlama, örnekleme dahil etme ya da dışarıda bırakma kriterlerinin katılığı, kontrol grubu olarak bekleme listesi kullanımı, seçkisizleştirme ve körleştirme sorunları, çalışmayı bırakan katılımcıların analizlere dahil edilmemesi ve psikoterapist etkilerinin ihmal edilmesi gibi problemler ve yanlılıklar içermektedir. Meta-analizler ile ilişkili başlıca sorunlar ise yayın yanlılığı ve düşük kaliteli çalışmaların meta-analizlere dahil edilmesidir. Tüm bu durumlar psikoterapilerin olumlu etkilerini olduğundan daha yüksek gösterirken, edinilen bulguların sağlıklı bir şekilde değerlendirilmesini engellemektedir. Uzun yıllardır gerçekleştirilen çalışmalar, psikoterapilerin çeşitli psikolojik rahatsızlıkların tedavisinde tercih edilebilecek etkili araçlar olduğunu göstermektedir. Ancak dikkat edilmesi gereken iki önemli husus vardır. Bunlardan ilki psikoterapilerin etkililiğini gösteren çalışmaların yöntemsel anlamda güncel bilimsel standartları yakalamakta zorlanmasıdır. İkinci önemli husus ise psikoterapilerin gerçek etkilerinin çalışmalarda bulunan etkilerden daha az olabileceğidir. Bu derleme makalesinin amacı, psikoterapi etkililik çalışmalarını ve meta-analizleri etkileyen yöntemsel problemlere ve onların çözümlerine değinerek klinik psikoloji alanında bu konuda ortaya çıkan farkındalığın arttırılmasına ve gerçekleştirilecek çalışmaların yöntemsel olarak güçlendirilmesine yardımcı olmaktır.

Keywords

• Psikoterapi
• tekrarlama krizi
• seçkisiz kontrollü çalışma
• meta-analiz
• yanlılık
• dodo kuşu yargısı

References

1. Abraham, W. T., ve Russell, D. W. (2008). Statistical power analysis in psychological research. Social and Personality Psychology Compass, 2(1), 283-301. https://doi.org/10.1111/j.1751-9004.2007.00052.x google scholar
2. Addis, M. E. ve Krasnow, A. D. (2000). A national survey of practicing psychologists’ attitudes toward psychotherapy treatment manuals. Journal of Consulting and Clinical Psychology, 68(2), 331-339. https://doi.org/10.1037/0022-006X.68.2.331 google scholar
3. APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. The American Psychologist, 61(4), 271-285. https://doi.org/10.1037/0003-066X.61.4.271 google scholar
4. Arnberg, A. ve Öst, L.-G. (2014). CBT for children with depressive symptoms: A meta-analysis. Cognitive Behaviour Therapy, 43(4), 275-288. https://doi.org/10.1080/16506073.2014.947316 google scholar
5. Baardseth, T. P., Goldberg, S. B., Pace, B. T., Wislocki, A. P., Frost, N. D., Siddiqui, J. R., Lindemann, A. M., Kivlighan, D. M. III, Laska, K. M., Del Re, A. C., Minami, T., & Wampold, B. E. (2013). Cognitive-behavioral therapy versus other therapies: Redux. Clinical Psychology Review, 33(3), 395-405. https://doi.org/10.1016Zj.cpr.2013.01.004 google scholar
6. Berkeljon, A., & Baldwin, S. A. (2009). An introduction to meta-analysis for psychotherapy outcome research. Psychotherapy Research, 19(4-5), 511-518. https://doi.org/10.1080/10503300802621172 google scholar
7. Bradley, H. A., Rucklidge, J. J. ve Mulder, R. T. (2017). A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials. Acta Psychiatrica Scandinavica, 135(1), 65-77. https://doi.org/10.1111/acps.12647 google scholar
8. Budd, R. ve Hughes, I. (2009). The Dodo Bird Verdict—controversial, inevitable and important: A commentary on 30 years of meta-analyses. Clinical Psychology & Psychotherapy, 16(6), 510-522. https://doi.org/10.1002/cpp.648 google scholar

9. Button, K. S., loannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J. ve Munafö, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. https://doi.org/10.1038/nrn3475 google scholar
10. Carroll, H. A., Toumpakari, Z., Johnson, L. ve Betts, J. A. (2017). The perceived feasibility of methods to reduce publication bias. PloS One, 12(10), e0186472. https://doi.org/10.1371/journal. pone.0186472 google scholar
11. Chambless, D. L. ve Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7-18. https://doi.org/10.1037/0022-006X.66.1.7 google scholar
12. Christley, R. M. (2010). Power and Error: Increased Risk of False Positive Results in Underpowered Studies. The Open Epidemiology Journal, 3(1), 16-19. http://dx.doi. org/10.2174/1874297101003010016 google scholar
13. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. https://doi. org/10.1037//0033-2909.112.1.155 google scholar
14. Cooper, A. A. ve Conklin, L. R. (2015). Drop out from individual psychotherapy for major depression: A meta-analysis of randomized clinical trials. Clinical Psychology Review, 40, 57-65. https://doi. org/10.1016/j.cpr.2015.05.001 google scholar
15. Coyne, J. C. ve Kok, R. N. (2014). Salvaging psychotherapy research: A manifesto. Journal of Evidence-Based Psychotherapies, 14(2), 105-124. google scholar
16. Craigie, M. A. ve Nathan, P. (2009). A nonrandomized effectiveness comparison of broad-spectrum group CBT to individual CBT for depressed outpatients in a community mental health setting. Behavior therapy, 40(3), 302-314. https://doi.org/10.1016/j.beth.2008.08.002 google scholar
17. Crameri, A., von Wyl, A., Koemeda, M., Schulthess, P. ve Tschuschke, V. (2015). Sensitivity analysis in multiple imputation in effectiveness studies of psychotherapy. Frontiers in Psychology, 6, 1042. https://doi.org/10.3389/fpsyg.2015.01042 google scholar
18. Cristea, I. A., Stefan, S., Karyotaki, E., David, D., Hollon, S. D. ve Cuijpers, P. (2017). The effects of cognitive behavioral therapy are not systematically falling: A revision of Johnsen and Friborg (2015). Psychological Bulletin, 143(3), 326-340. https://doi.org/10.1037/bul0000062 google scholar
19. Cuijpers, P. ve Cristea, I. A. (2016). How to prove that your therapy is effective, even when it is not: A guideline. Epidemiology and Psychiatric Sciences, 25(5), 428-435. https://doi.org/10.1017/ S2045796015000864 google scholar
20. Cuijpers, P., Cristea, I. A., Karyotaki, E., Reijnders, M. ve Huibers, M. J. H. (2016). How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence. World Psychiatry, 15(3), 245-258. https://doi.org/10.1002/wps.20346 google scholar
21. Cuijpers, P., Karyotaki, E., Reijnders, M. ve Ebert, D. D. (2018). Was Eysenck right after all? A reassessment of the effects of psychotherapy for adult depression. Epidemiology and Psychiatric Sciences, 28(1), 21-30. https://doi.org/10.1017/S2045796018000057 google scholar
22. Cuijpers, P., Karyotaki, E., Reijnders, M. ve Ebert, D. D. (2019). Is psychotherapy effective? Pretending everything is fine will not help the field forward. Epidemiology and Psychiatric Sciences, 28(3), 356-357. https://doi.org/10.1017/S204579601800080X google scholar
23. Cuijpers, P., Reijnders, M. ve Huibers, M. J. H. (2019). The Role of Common Factors in Psychotherapy Outcomes. Annual Review of Clinical Psychology, 15(1), 207-231. https://doi.org/10.1146/annurev-clinpsy-050718-095424 google scholar
24. Cuijpers, P., van Straten, A., Bohlmeijer, E., Hollon, S. D., & Andersson, G. (2010). The effects of psychotherapy for adult depression are overestimated: A meta-analysis of study quality and effect size. PsychologicalMedicine, 40(2), 211-223. https://doi.org/10.1017/S0033291709006114 google scholar
25. de Felice, G., Giuliani, A., Halfon, S., Andreassi, S., Paoloni, G. ve Orsucci, F. F. (2019). The misleading Dodo Bird verdict. How much of the outcome variance is explained by common and specific factors? New Ideas in Psychology, 54, 50-55. https://doi.org/10.1016/j.newideapsych.2019.01.006 google scholar
26. Del Re, A. C., Flückiger, C., Horvath, A. O., Symonds, D. ve Wampold, B. E. (2012). Therapist effects in the therapeutic alliance-outcome relationship: A restricted-maximum likelihood meta-analysis. Clinical Psychology Review, 32(7), 642-649. https://doi.org/10.1016/j.cpr.2012.07.002 google scholar
27. Donnon, T. (2012). Experimental or RCT research designs: A crisis of nomenclature in medical education. Canadian Medical Education Journal, 3(2), 82-84. google scholar
28. Driessen, E., Hollon, S. D., Bockting, C. L. H., Cuijpers, P. ve Turner, E. H. (2015). Does Publication Bias Inflate the Apparent Efficacy of Psychological Treatment for Major Depressive Disorder? A Systematic Review and Meta-Analysis of US National Institutes of Health-Funded Trials. PloS One, 10(9), e0137864. https://doi.org/10.1371/journal.pone.0137864 google scholar
29. Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F. ve Munafö, M. R. (2017). Low statistical power google scholar
30. in biomedical science: A review of three human research domains. Royal Society Open Science, 4(2), 160254. https://doi.org/10.1098/rsos.160254 google scholar
31. Easterbrook, P. J., Gopalan, R., Berlin, J. A. ve Matthews, D. R. (1991). Publication bias in clinical research. The Lancet, 337(8746), 867-872. https://doi.org/10.1016/0140-6736(91)90201-Y google scholar
32. Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16(5), 319-324. https://doi.org/10.1037/h0063633 google scholar
33. Eysenck, H. J. (1964). The outcome problem in psychotherapy: A reply. Psychotherapy: Theory, Research & Practice, 1(3), 97-100. https://doi.org/10.1037/h0088591 google scholar
34. Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891-904. https://doi.org/10.1007/s11192-011-0494-7 google scholar
35. Faul, F., Erdfelder, E., Lang, A.-G., ve Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191. https://doi.org/10.3758/BF03193146 google scholar
36. Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532-538. https://doi.org/10.1037/a0015808 google scholar
37. Fensterheim, H. ve Raw, S. D. (1996). Psychotherapy Research Is Not Psychotherapy Practice. Clinical Psychology: Science and Practice, 3(2), 168-171. https://doi.org/10.1111/j.1468-2850.1996. tb00067.x google scholar
38. Fernandez, E., Salem, D., Swift, J. K. ve Ramtahal, N. (2015). Meta-analysis of dropout from cognitive behavioral therapy: Magnitude, timing, and moderators. Journal of Consulting and Clinical Psychology, 83(6), 1108-1122. https://doi.org/10.1037/ccp0000044 google scholar
39. Flint, J., Cuijpers, P., Horder, J., Koole, S. L. ve Munafö, M. R. (2015). Is there an excess of significant findings in published studies of psychotherapy for depression? Psychological Medicine, 45(2), 439446. https://doi.org/10.1017/S0033291714001421 google scholar
40. Fonagy, P., Luyten, P. ve Bateman, A. (2017). Treating Borderline Personality Disorder With Psychotherapy: Where Do We Go From Here? JAMA Psychiatry, 74(4), 316-317. https://doi. org/10.1001/jamapsychiatry.2016.4302 google scholar
41. Freire, E. S. (2006). Randomized Controlled Clinical Trial in Psychotherapy Research: An Epistemological Controversy. Journal of Humanistic Psychology, 46(3), 323-335. https://doi. org/10.1177/0022167806286276 google scholar
42. Friborg, O. ve Johnsen, T. J. (2017). The effect of cognitive-behavioral therapy as an antidepressive treatment is falling: Reply to Ljotsson et al. (2017) and Cristea et al. (2017). Psychological Bulletin, 143(3), 341-345. https://doi.org/10.1037/bul0000090 google scholar
43. Furukawa, T. A., Noma, H., Caldwell, D. M., Honyashiki, M., Shinohara, K., Imai, H., Chen, P., Hunot, V., & Churchill, R. (2014). Waiting list may be a nocebo condition in psychotherapy trials: A contribution from network meta-analysis. Acta Psychiatrica Scandinavica, 130(3), 181192. https://doi.org/10.1111/acps.12275 google scholar
44. Gelman, A. ve Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641-651. https://doi. org/10.1177/1745691614551642 google scholar
45. Gelo, O. C. G., Lagetto, G., Dinoi, C., Belfiore, E., Lombi, E., Blasi, S., Aria, M., & Ciavolino, E. (2020). Which methodological practice(s) for psychotherapy science? A systematic review and a proposal. Integrative Psychological & Behavioral Science, 54(1), 215-248. https://doi.org/10.1007/ s12124-019-09494-3 google scholar
46. Glass, G. V. (1982). Meta-analysis: An Approach to the Synthesis of Research Results. Journal of research in science teaching, 19(2), 93-112. google scholar
47. Goldfried, M. R. (2020). The field of psychotherapy: Over 100 years old and still an infant science. Clinical Psychology in Europe, 2(1), e2753. https://doi.org/10.32872/cpe.v2i1.2753 google scholar
48. Goldfried, M. R. ve Wolfe, B. E. (1998). Toward a more clinically valid approach to therapy research. Journal of Consulting and Clinical Psychology, 66(1), 143-150. https://doi.org/10.1037/0022-006X.66.1.143 google scholar
49. Gonzalez-Blanch, C. ve Carral-Fernandez, L. (2017). Cage up Dodo, please! The tale of all psychotherapies being equally effective. Papeles del Psicologo, 38(2), 94-106. https://doi. org/10.23923/pap.psicol2017.2828 google scholar
50. Hansen, W. B. ve Collins, L. M. (1994). Seven ways to increase power without increasing N. NIDA Research Monograph, 142, 184-195. google scholar
51. Hengartner, M. P. (2018). Raising awareness for the replication crisis in clinical psychology by focusing on inconsistencies in psychotherapy research: How much can we rely on published findings from efficacy trials? Frontiers in Psychology, 9, Article 256. https://doi.org/10.3389/fpsyg.2018.00256 google scholar
52. Hoertel, N., de Maricourt, P., Katz, J., Doukhan, R., Lavaud, P., Peyre, H. ve Limosin, F. (2014). Are participants in pharmacological and psychotherapy treatment trials for social anxiety disorder representative of patients in real-life settings? Journal of Clinical Psychopharmacology, 34(6), 697-703. https://doi.org/10.1097/JCP.0000000000000204 google scholar
53. Hoertel, N., Lopez, S., Wang, S., Gonzalez-Pinto, A., Limosin, F. ve Blanco, C. (2015). Generalizability of pharmacological and psychotherapy clinical trial results for borderline personality disorder to community samples. Personality Disorders: Theory, Research, and Treatment, 6(1), 81-87. https:// doi.org/10.1037/per0000091 google scholar
54. Hopewell, S., Clarke, M. ve Mallett, S. (2005). Grey Literature and Systematic Reviews. H. R. Rothstein & A. J. Sutton (Ed.), Publication Bias in Meta-Analysis içinde (s. 49-72). John Wiley & Sons, Ltd. https://doi.org/10.1002/0470870168.ch4 google scholar
55. Hotopf, M. (2002). The pragmatic randomised controlled trial. Advances in Psychiatric Treatment, 8(5), 326-333. doi:10.1192/apt.8.5.326 google scholar
56. Hsu, L. M. (1992). Random sampling, randomization, and equivalence of contrasted groups in psychotherapy outcome research. A. E. Kazdin (Ed.), Methodological issues & strategies in clinical research içinde (s. 91-105). American Psychological Association. https://doi.org/10.1037/10109-018 google scholar
57. Johns, R. G., Barkham, M., Kellett, S. ve Saxon, D. (2019). A systematic review of therapist effects: A critical narrative update and refinement to Baldwin and Imel’s (2013) review. Clinical Psychology Review, 67, 78-93. https://doi.org/10.1016/j.cpr.2018.08.004 google scholar
58. Johnsen, T. J. ve Friborg, O. (2015). The effects of cognitive behavioral therapy as an anti-depressive treatment is falling: A meta-analysis. Psychological Bulletin, 141(4), 747-768. https://doi. org/10.1037/bul0000015 google scholar
59. Jones, J. L. ve Mehr, S. L. (2007). Foundations and assumptions of the scientist-practitioner model. American Behavioral Scientist, 50(6), 766-771. https://doi.org/10.1177/0002764206296454 google scholar
60. Kelley, K. ve Preacher, K. J. (2012). On effect size. Psychological Methods, 17(2), 137-152. https://doi. org/10.1037/a0028086 google scholar
61. Kendall, J. (2003). Designing a research project: Randomised controlled trials and their principles. Emergency Medicine Journal: EMJ, 20(2), 164-168. https://dx.doi.org/10.1136%2Femj.20.2.164 google scholar
62. Kiai, A. (2019). To protect credibility in science, banish “publish or perish”. Nature Human Behaviour, 3(10), 1017-1018. https://doi.org/10.1038/s41562-019-0741-0 google scholar
63. Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R. ve Williamson, P. R. (2010). The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 340, c365. https://doi.org/10.1136/bmj.c365 google scholar
64. Lambert, M. J. (2013). The Efficacy and Effectiveness of Psychotherapy. M. J. Lambert (Ed.), Bergin and Garfields Handbook of Psychotherapy and Behavior Change (6. Bs) içinde (s. 169-218). John Wiley & Sons, Ltd. google scholar
65. Landman, J. T. ve Dawes, R. M. (1982). Psychotherapy outcome: Smith and Glass’ conclusions stand up under scrutiny. American Psychologist, 37(5), 504-516. https://doi.org/10.1037/0003-066X.37.5.504 google scholar
66. Laska, K. M., Gurman, A. S. ve Wampold, B. E. (2014). Expanding the lens of evidence-based practice in psychotherapy: A common factors perspective. Psychotherapy, 51(4), 467-481. https://doi. org/10.1037/a0034332 google scholar
67. Leichsenring, F., Abbass, A., Hilsenroth, M. J., Leweke, F., Luyten, P., Keefe, J. R., Midgley, N., Rabung, S., Salzer, S. ve Steinert, C. (2017). Biases in research: Risk factors for non-replicability in psychotherapy and pharmacotherapy research. Psychological Medicine, 47(6), 1000-1011. https:// doi.org/10.1017/S003329171600324X google scholar
68. Leichsenring, F., Abbass, A., Hilsenroth, M. J., Luyten, P., Munder, T., Rabung, S. ve Steinert, C. (2018). “Gold standards,” plurality and monocultures: The need for diversity in psychotherapy. Frontiers in Psychiatry, 9, 159. https://doi.org/10.3389/fpsyt.2018.00159 google scholar
69. Leichsenring, F. ve Rabung, S. (2011). Long-term psychodynamic psychotherapy in complex mental disorders: Update of a meta-analysis. The British Journal of Psychiatry, 199(1), 15-22. https://doi. org/10.1192/bjp.bp.110.082776 google scholar
70. Leykin, Y. ve DeRubeis, R. J. (2009). Allegiance in Psychotherapy Outcome Research: Separating Association From Bias. Clinical Psychology: Science and Practice, 16(1), 54-65. https://doi. org/10.1111/j.1468-2850.2009.01143.x google scholar
71. Lilienfeld, S. O. (2014). The Dodo Bird verdict: Status in 2014. The Behavior Therapist, 37(4), 91-95. google scholar
72. Linardon, J., Kothe, E. J. ve Fuller-Tyszkiewicz, M. (2019). Efficacy of psychotherapy for bulimia nervosa and binge-eating disorder on self-esteem improvement: Meta-analysis. European Eating Disorders Review, 27(2), 109-123. https://doi.org/10.1002/erv.2662 google scholar
73. Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., Halperin, G., Bishop, M., Berman, J. S., & Schweizer, E. (1999). The researcher’s own therapy allegiances: A “wild card” in comparisons of treatment efficacy. Clinical Psychology: Science and Practice, 6(1), 95-106. https://doi.Org/10.1093/clipsy.6.1.95 google scholar
74. Luborsky, L., Rosenthal, R., Diguer, L., Andrusyna, T. P., Berman, J. S., Levitt, J. T., Seligman, D. A., & Krause, E. D. (2002). The dodo bird verdict is alive and well--mostly. Clinical Psychology: Science and Practice, 9(1), 2-12. https://doi.org/10.1093/clipsy.9.1.2 google scholar
75. Luborsky, L., Singer, B. ve Luborsky, L. (1975). Comparative studies of psychotherapies: Is it true that “everyone has won and all must have prizes”? Archives of General Psychiatry, 32(8), 995-1008. https://doi.org/10.1001/archpsyc.1975.01760260059004 google scholar
76. Marcus, D. K., O’Connell, D., Norris, A. L. ve Sawaqdeh, A. (2014). Is the Dodo bird endangered in the 21st century? A meta-analysis of treatment comparison studies. Clinical Psychology Review, 34(7), 519-530. https://doi.org/10.1016/j.cpr.2014.08.001 google scholar
77. Maxwell, S. E., Lau, M. Y. ve Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487-498. https://doi. org/10.1037/a0039400 google scholar
78. Miguel, C., Karyotaki, E., Cuijpers, P. ve Cristea, I. A. (2021). Selective outcome reporting and the effectiveness of psychotherapies for depression. World Psychiatry, 20(3), 444-445. https://dx.doi. org/10.1002%2Fwps.20900 google scholar
79. Miller, L. E., ve Stewart, M. E. (2011). The blind leading the blind: Use and misuse of blinding in randomized controlled trials. Contemporary Clinical Trials, 32(2), 240-243. https://doi. org/10.1016/j.cct.2010.11.004 google scholar
80. Morrison, K. H., Bradley, R. ve Westen, D. (2003). The external validity of controlled clinical trials of psychotherapy for depression and anxiety: A naturalistic study. Psychology and Psychotherapy: Theory, Research and Practice, 76(2), 109-132. https://doi.org/10.1348/147608303765951168 google scholar
81. Mulder, R., Singh, A. B., Hamilton, A., Das, P., Outhred, T., Morris, G., Bassett, D., Baune, B. T., Berk, M., Boyce, P., Lyndon, B., Parker, G., & Malhi, G. S. (2018). The limitations of using randomised controlled trials as a basis for developing treatment guidelines. Evidence-based mental health, 21 (1), 4-6. https://doi.org/10.1136/eb-2017-102701 google scholar
82. Munder, T., Brütsch, O., Leonhart, R., Gerger, H. ve Barth, J. (2013). Researcher allegiance in psychotherapy outcome research: An overview of reviews. Clinical Psychology Review, 33(4), 501-511. https://doi.org/10.1016/j.cpr.2013.02.002 google scholar Munder, T., Flückiger, C., Leichsenring, F., Abbass, A. A., Hilsenroth, M. J., Luyten, P., Rabung, S., Steinert, C., & Wampold, B. E. (2019). Is psychotherapy effective? A re-analysis of treatments for depression. Epidemiology and psychiatric sciences, 28(3), 268-274. https://doi.org/10.1017/ S2045796018000355 google scholar
83. Münch, R. (2014). Academic capitalism: Universities in the global struggle for excellence. Routledge. google scholar
84. Nahum, D., Alfonso, C. A. ve Sönmez, E. (2019). Common Factors in Psychotherapy. A. Javed ve K. N. Fountoulakis (Ed.), Advances in Psychiatry içinde (s. 471-481). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-70554-5_29 google scholar
85. Niemeyer, H., Musch, J. ve Pietrowsky, R. (2012). Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for schizophrenia. Schizophrenia Research, 138(2), 103-112. https://doi.org/10.1016/j.schres.2012.03.023 google scholar
86. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., Ishiyama, J., ... Yarkoni, T. (2015). SCIENTIFIC STANDARDS. Promoting an open research culture. Science (New York, N.Y.), 348(6242), 1422-1425. https://doi.org/10.1126/science.aab2374 google scholar
87. Open Science Collaboration. (2015). PSYCHOLOGY. Estimating the reproducibility of psychological science. Science, 349, Article aac4716. https://doi.org/10.1126/science.aac4716 google scholar
88. O’rourke, K. (2007). An historical perspective on meta-analysis: Dealing quantitatively with varying study results. Journal of the Royal Society of Medicine, 100(12), 579-582. https://doi. org/10.1177/0141076807100012020 google scholar
89. Owen, J., Drinane, J. M., Idigo, K. C. ve Valentine, J. C. (2015). Psychotherapist effects in meta-analyses: How accurate are treatment effects? Psychotherapy, 52(3), 321-328. https://doi. org/10.1037/pst0000014 google scholar
90. Peng, C.-Y. J. ve Chen, L.-T. (2014). Beyond Cohen’s d: Alternative Effect Size Measures for Between-Subject Designs. The Journal of Experimental Education, 82(1), 22-50. https://doi.org/10.1080/00 220973.2012.745471 google scholar
91. Persons, J. B. (1991). Psychotherapy outcome studies do not accurately represent current models of psychotherapy: A proposed remedy. American Psychologist, 46(2), 99-106. https://doi.org/10.1037/0003-066X.46.2.99 google scholar
92. Persons, J. B., Roberts, N. A., Zalecki, C. A. ve Brechwald, W. A. (2006). Naturalistic outcome of case formulation-driven cognitive-behavior therapy for anxious depressed outpatients. Behaviour Research and Therapy, 44(7), 1041-1051. https://doi.org/10.1016/j.brat.2005.08.005 google scholar
93. Persons, J. B. ve Silberschatz, G. (1998). Are results of randomized controlled trials useful to psychotherapists? Journal of Consulting and Clinical Psychology, 66(1), 126-135. https://doi. org/10.1037/0022-006X.66.1.126 google scholar
94. Philips, B., & Falkenström, F. (2021). What Research Evidence Is Valid for Psychotherapy Research?. Frontiers in psychiatry, 11, 625380. https://doi.org/10.3389/fpsyt.2020.625380 google scholar
95. Rifkin, A. (2007). Randomized controlled trials and psychotherapy research [Editorial]. The American Journal of Psychiatry, 164(1), 7-8. https://doi.Org/10.1176/appi.ajp.164.1.7 google scholar
96. Rosenzweig, S. (1954). A transvaluation of psychotherapy: A reply to Hans Eysenck. The Journal of Abnormal and Social Psychology, 49(2), 298-304. https://doi.org/10.1037/h0061172 google scholar
97. Rossi, J. S. (2013). Statistical power analysis. J. A. Schinka, W. F. Velicer ve I. B. Weiner (Ed.), Handbook ofpsychology: Research methods in psychology içinde (s. 71-108). John Wiley & Sons, Inc.. google scholar
98. Rothstein, H. R. ve Hopewell, S. (2009). Grey literature. H. Cooper, L. V. Hedges ve J. C. Valentine (Ed.), The handbook of research synthesis and meta-analysis içinde (s. 103-125). Russell Sage Foundation. google scholar
99. Rubio-Aparicio, M., Marin-Martinez, F., Sanchez-Meca, J. ve L6pez-L6pez, J. A. (2018). A methodological review of meta-analyses of the effectiveness of clinical psychology treatments. Behavior Research Methods, 50(5), 2057-2073. https://doi.org/10.3758/s13428-017-0973-8 google scholar
100. Sakaluk, J. K., Williams, A. J., Kilshaw, R. E. ve Rhyner, K. T. (2019). Evaluating the evidential value of empirically supported psychological treatments (ESTs): A meta-scientific review. Journal of Abnormal Psychology, 128(6), 500-509. https://doi.org/10.1037/abn0000421 google scholar
101. Schafer, T. ve Schwarz, M. A. (2019). The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.00813 google scholar
102. Shapiro, D. A. ve Shapiro, D. (1982). Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychological Bulletin, 92(3), 581-604. https://doi.org/10.1037/0033-2909.92.3.581 google scholar
103. Shean, G. (2012). Some limitations on the external validity of psychotherapy efficacy studies and suggestions for future research. American Journal of Psychotherapy, 66(3), 227-242. https://doi. org/10.1176/appi.psychotherapy.2012.66.3.227 google scholar
104. Shean, G. (2016). Psychotherapy Outcome Research: Issues and Questions. Psychodynamic Psychiatry, 44(1), 1-24. https://doi.org/10.1521/pdps.2016.44.1.1 google scholar
105. Shedler, J. (2018). Where Is the Evidence for “Evidence-Based” Therapy? Psychiatric Clinics ofNorth America, 41(2), 319-329. https://doi.org/10.1016/j.psc.2018.02.001 google scholar
106. Shelby, L. B. ve Vaske, J. J. (2008). Understanding meta-analysis: A review of the methodological literature. Leisure Sciences, 30(2), 96-110. https://doi.org/10.1080/01490400701881366 google scholar
107. Shinohara, K., Tajika, A., Imai, H., Takeshima, N., Hayasaka, Y. ve Furukawa, T. A. (2015). Protocol registration and selective outcome reporting in recent psychiatry trials: New antidepressants and cognitive behavioural therapies. Acta Psychiatrica Scandinavica, 132(6), 489-498. https://doi. org/10.1111/acps.12502 google scholar
108. Smit, Y., Huibers, M. J. H., Ioannidis, J. P. A., van Dyck, R., van Tilburg, W. ve Arntz, A. (2012). google scholar
109. The effectiveness of long-term psychoanalytic psychotherapy—A meta-analysis of randomized controlled trials. Clinical Psychology Review, 32(2), 81-92. https://doi.org/10.1016/j. cpr.2011.11.003 google scholar
110. Smith, M. L. ve Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752-760. https://doi.org/10.1037/0003-066X.32.9.752 google scholar
111. Sohn, D. (1996). Publication bias and the evaluation of psychotherapy efficacy in reviews of the research literature. Clinical Psychology Review, 16(2), 147-156. https://doi.org/10.1016/0272-7358(96)00005-0 google scholar
112. Remission in CBT for adult anxiety disorders: A meta-analysis. Clinical Psychology Review, 61, 1-8. https://doi.org/10.1016/j.cpr.2018.03.002 google scholar
113. Stricker, J. ve Günther, A. (2019). Scientific misconduct in psychology: A systematic review of prevalence estimates and new empirical data. Zeitschrift für Psychologie, 227(1), 53-63. http:// dx.doi.org/10.1027/2151-2604/a000356 google scholar
114. Strupp, H. H. (1963). The outcome problem in psychotherapy revisited. Psychotherapy: Theory, Research & Practice, 1(1), 1-13. https://doi.org/10.1037/h0088565 google scholar
115. Tackett, J. L., Brandes, C. M., King, K. M. ve Markon, K. E. (2019). Psychology’s replication crisis and clinical psychological science. Annual Review of Clinical Psychology, 15, 579-604. https://doi. org/10.1146/annurev-clinpsy-050718-095710 google scholar
116. Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017). It’s time to broaden the replicability conversation: Thoughts for and from clinical psychological science. Perspectives on PsychologicalScience, 12(5), 742-756. https://doi. org/10.1177/1745691617690042 google scholar
117. Task Force on Promotion and Dissemination of Psychological Procedures Division of Clinical Psychology. (1995). Training in and dissemination of empirically-validated psychological treatments: Report and recommendations. The Clinical Psychologist, 48(1), 3-23. google scholar
118. Thase, M. E. (1999). What is the investigator allegiance effect and what should we do about it? Clinical Psychology: Science and Practice, 6(1), 113-115. https://doi.org/10.1093/clipsy.6.1.113 google scholar
119. Tracey, T. J. G., Wampold, B. E., Lichtenberg, J. W. ve Goodyear, R. K. (2014). Expertise in psychotherapy: An elusive goal? American Psychologist, 69(3), 218-229. https://doi.org/10.1037/ a0035099 google scholar
120. Tolin, D. F. (2010). Is cognitive-behavioral therapy more effective than other therapies?: A meta-analytic review. Clinical Psychology Review, 30(6), 710-720. https://doi.org/10.1016/j.cpr.2010.05.003 google scholar
121. Tolin, D. F., McKay, D., Forman, E. M., Klonsky, E. D. ve Thombs, B. D. (2015). Empirically supported treatment: Recommendations for a new model. Clinical Psychology: Science and Practice, 22(4), 317-338. https://doi.org/10.1111/cpsp.12122 google scholar
122. van der Zweerde, T., Bisdounis, L., Kyle, S. D., Lancee, J. ve van Straten, A. (2019). Cognitive behavioral therapy for insomnia: A meta-analysis of long-term effects in controlled studies. Sleep Medicine Reviews, 48, 101208. https://doi.org/10.1016/j.smrv.2019.08.002 google scholar
123. Vevea, J. L. ve Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10(4), 428-443. google scholar
124. von Wolff, A., Jansen, M., Hölzel, L. P., Westphal, A., Harter, M. ve Kriston, L. (2014). Generalizability of findings from efficacy trials for chronic depression: An analysis of eligibility criteria. Psychiatric Services, 65(7), 897-904. https://doi.org/10.1176/appi.ps.201300309 google scholar
125. Wampold, B. E. (2001). The great psychotherapy debate: Models, methods, and findings. Routledge. google scholar
126. Wampold, B. E. (2015). How important are the common factors in psychotherapy? An update. World Psychiatry, 14(3), 270-277. https://doi.org/10.1002/wps.20238 google scholar
127. Wampold, B. E. (2019). The basics of psychotherapy: An introduction to theory and practice (2. Bs.). American Psychological Association. https://doi.org/10.1037/0000117-000 google scholar
128. Wampold, B. E., Hollon, S. D. ve Hill, C. E. (2011). Unresolved questions and future directions in psychotherapy research. History of psychotherapy: Continuity and change (2. Bs) içinde (s. 333356). American Psychological Association. https://doi.org/10.1037/12353-011 google scholar
129. Wampold, B. E., Mondin, G. W., Moody, M., Stich, F., Benson, K. ve Ahn, H. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empiricially, “all must have prizes.” Psychological Bulletin, 122(3), 203-215. https://doi.org/10.1037/0033-2909.122.3.203 google scholar
130. Weinberger, J. (1995). Common factors aren’t so common: The common factors dilemma. Clinical Psychology: Science and Practice, 2(1), 45-69. https://doi.Org/10.1111/j.1468-2850.1995. tb00024.x google scholar
131. Westbrook, D. ve Kirk, J. (2005). The clinical effectiveness of cognitive behaviour therapy: Outcome for a large sample of adults treated in routine practice. Behaviour Research and Therapy, 43(10), 1243-1261. https://doi.org/10.1016/j.brat.2004.09.006 google scholar
132. Westen, D. ve Bradley, R. (2005). Empirically supported complexity: Rethinking evidence-based practice in psychotherapy. Current Directions in Psychological Science, 14(5), 266-271. https:// doi.org/10.1111/j.0963-7214.2005.00378.x google scholar
133. Westen, D. ve Morrison, K. (2001). A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: An empirical examination of the status of empirically supported therapies. Journal of Consulting and Clinical Psychology, 69(6), 875-899. https://doi. org/10.1037/0022-006X.69.6.875 google scholar
134. Westen, D., Novotny, C. M. ve Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130(4), 631-663. https://doi.org/10.1037/0033-2909.130.4.631 google scholar
135. Wierzbicki, M. ve Pekarik, G. (1993). A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice, 24(2), 190-195. https://doi.org/10.1037/0735-7028.24.2.190 google scholar
136. Wojnarowski, C., Firth, N., Finegan, M. ve Delgadillo, J. (2019). Predictors of depression relapse and recurrence after cognitive behavioural therapy: A systematic review and meta-analysis. Behavioural and Cognitive Psychotherapy, 47(5), 514-529. https://doi.org/10.1017/S1352465819000080 google scholar