Creating a Clinical Psychology Dataset with Synthetic Data: Automatic Detection of Cognitive Distortions Classified with NLP

Öz

Cognitive distortions are thought errors that lead individuals to perceive reality in a misleading way and are strongly associated with psychopathologies. Therefore, accurately identifying and classifying distortions can enhance the effectiveness of cognitive-behavioral therapy (CBT). This study investigates the effectiveness of deep learning and NLP techniques for the automatic detection of cognitive distortions. The RoBERTa model was trained using English synthetic data generated by GPT-4 (2000 examples) and the dataset from Shreevastava and Foltz (1590 cognitive distortion examples, 933 non-distortion examples). Three scenarios were tested: the original dataset, the synthetic dataset, and their combination. The results showed that synthetic data is a strong resource. Accuracy rates were 60.67% (original), 94.51% (synthetic), and 77.18% (combined). The GPT-4-based dataset provided almost perfect F1 scores, particularly in some categories. ROC curve analyses showed that the GPT-4 dataset had the highest AUC value (0.80). The study revealed that using synthetic data expands the potential of AI applications in clinical psychology and offers a way to develop effective models while preserving patient privacy. Future research should test synthetic data with different models and compare it with real clinical data.

Anahtar Kelimeler

• Cognitive distortion
• ; machine learning
• natural language processing
• GPT-4
• depression

Sentetik Verilerle Klinik Psikoloji Veri Seti Oluşturma: Bilişsel Çarpıtmaların NLP ile Sınıflandırılarak Otomatik Tespiti

Öz

Bilişsel çarpıtmalar, bireylerin gerçekliği yanıltıcı bir şekilde algılamalarına neden olan düşünce hatalarıdır ve psikopatolojilerle güçlü bir ilişkisi vardır. Bu nedenle, çarpıtmaların doğru bir şekilde belirlenmesi ve sınıflandırılması, bilişsel davranışçı terapinin (CBT) etkinliğini artırabilir. Bu çalışma, bilişsel çarpıtmaların otomatik tespiti için derin öğrenme ve NLP tekniklerinin etkinliğini incelemektedir. GPT-4 ile üretilen İngilizce sentetik veriler (2000 örnek) ve Shreevastava ve Foltz’un veri seti (1590 bilişsel çarpıtma, 933 çarpıtma içermeyen örnek) kullanılarak RoBERTa modeli eğitilmiştir. Üç senaryo test edilmiştir: orijinal veri seti, sentetik veri seti ve bunların kombinasyonu. Sonuçlar, sentetik verilerin güçlü bir kaynak olduğunu göstermiştir. Doğruluk oranları sırasıyla %60,67 (orijinal), %94,51 (sentetik) ve %77,18 (kombine) olarak elde edilmiştir. GPT-4 tabanlı veri seti, özellikle bazı kategorilerde neredeyse mükemmel F1 skorları sağlamıştır. ROC eğrisi analizleri, GPT-4 veri setinin en yüksek AUC değerine (0,80) sahip olduğunu göstermiştir. Çalışma, sentetik veri kullanımının klinik psikolojide yapay zeka uygulamalarının potansiyelini genişlettiğini ve hasta gizliliğini korurken etkili modeller geliştirmenin bir yolunu sunduğunu ortaya koymuştur. Gelecekteki araştırmalar için, sentetik verilerin farklı modellerle test edilmesi ve gerçek klinik verilerle karşılaştırılması önerilmektedir.

Anahtar Kelimeler

• Bilişsel çarpıtma
• makine öğrenimi
• doğal dil işleme
• GPT-4
• depresyon

Kaynakça

1. Beck AT. Cognitive therapy and emotional disorders. New York: New American Library; 1976.
2. Ellis A. Reason and emotion in psychotherapy. New York: Lyle Stuart; 1962.
3. Burns DD. Feeling good: The new mood therapy. New York: New American Library; 1980.
4. Beck AT. Cognitive therapy: Nature and relation to behavior therapy. Behav Ther. 1970;1(2):184-200.
5. Rnic K, Dozois DJ, Martin RA. Cognitive distortions, humor styles, and depression. Eur J Psychol. 2016;12(3):348-62.
6. Marton P, Kutcher S. The prevalence of cognitive distortion in depressed adolescents. J Psychiatry Neurosci. 1995;20(1):33.
7. Attia E, Schroeder L. Pharmacologic treatment of anorexia nervosa: where do we go from here? Int J Eat Disord. 2005;37(S1):S60-S63.
8. Altuntaş Y, Söyler HÇ, Kula H. Kumar bağımlılarıyla sağlıklı kontrollerin bilişsel çarpıtmaları, psikopatolojileri ve aile ilişkilerinin karşılaştırılması. Sosyal Beşeri ve İdari Bilimler Dergisi. 2023;6(1):68-84.

9. Buga A, Kaya İ. The role of cognitive distortions related to academic achievement in predicting the depression, stress, and anxiety levels of adolescents. Int J Contemp Educ Res. 2022;9(1):103-14.
10. Zhou M, Duan N, Liu S, Shum H. Progress in neural NLP: Modeling, learning, and reasoning. Engineering. 2020.
11. Wang Y. Basic methodologies used in NLP area. 2020 IEEE 3rd International Conference on Automation, Electronics and Electrical Engineering (AUTEEE). 2020:505-11.
12. Tenney I, Das D, Pavlick E. BERT rediscovers the classical NLP pipeline. 2019:4593-601. https://doi.org/10.18653/v1/P19-1452.
13. Wallace E, Gardner M, Singh S. Interpreting predictions of NLP models. 2020:20-23. https://doi.org/10.18653/v1/2020.emnlp-tutorials.3.
14. Harrigian K, Aguirre C, Dredze M. On the state of social media data for mental health research. In: 7th Workshop on Computational Linguistics and Clinical Psychology: Improving Access, CLPsych 2021; 2021:15-24.
15. Shickel B, Siegel S, Heesacker M, Benton S, Rashidi P. Automatic detection and classification of cognitive distortions in mental health text. 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE). 2020:275-80.
16. Kaggle. Therapist QA dataset [Internet]. Available from: https://www.kaggle.com/datasets/arnmaud/therapist-qa. Accessed 30 Oct 2024.
17. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
18. Shreevastava S, Foltz PW. Detecting cognitive distortions from patient-therapist interactions. NAACL HLT. 2021;151.
19. Kaggle.Cognitive Distortion Detection Dataset [Internet]. Available from: https://www.kaggle.com/datasets/sagarikashreevastava/cognitive-distortion-detetction-dataset?select=Annotated_data.csv. Accessed 30 Oct 2024.
20. OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
21. Brown T, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877-901.
22. Gao L, Zhang L, Zhang L, Huang J. RSVN: A RoBERTa sentence vector normalization scheme for short texts to extract semantic information. Appl Sci. 2022;12(21):11278.
23. ChatGPT. Shared conversation [Internet]. Available from: https://chatgpt.com/share/e/e7f0600a-68c3-482d-8647-a40cc4b8ee7a. Accessed 30 Oct 2024.
24. HuggingFace. Dataset-1 [Internet]. Available from: https://doi.org/10.57967/hf/2858. Accessed 30 Oct 2024.
25. HuggingFace. Dataset-2 [Internet]. Available from: https://doi.org/10.57967/hf/2857. Accessed 30 Oct 2024.
26. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
27. Chung J, Gulcehre C, Cho K, Bengio Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
28. Zhang X, Zhao J, LeCun Y. A novel RoBERTa-GRU model for sentiment analysis. Appl Sci. 2023;13(6):3915.
29. HuggingFace. Model-1 [Internet]. Available from: https://doi.org/10.57967/hf/2859. Accessed 30 Oct 2024.
30. HuggingFace. Model-2 [Internet]. Available from: https://doi.org/10.57967/hf/2832. Accessed 30 Oct 2024.