Psychometric and Explainable Artificial Intelligence–Supported Machine Learning Analysis of Social Media Risk Levels

Abstract

The aim of this study is to classify self-reported risk levels associated with social media use in a reliable and transparent manner. To this end, a five-level risk classification was evaluated through the integration of psychometric indicators, machine learning (ML)–based classification, and explainable artificial intelligence (XAI) approaches. The analysis is based on Likert-type items scaled between 1 and 5 (1 = lowest, 5 = highest). Within the framework of exploratory factor analysis (EFA), the results of the Kaiser–Meyer–Olkin (KMO = 0.862), Bartlett’s test of sphericity (P<0.001), Cronbach’s alpha (α = 0.863), and the eigenvalue ratio (EV₁/EV₂ = 3.71) indicated that the scale is unidimensional and internally consistent. A Random Forest (RF) model was applied for multiclass prediction, and its probability estimates were calibrated using isotonic regression. Validation results yielded ROC-AUC = 0.959, PR-AUC = 0.868, Brier = 0.255, and LogLoss = 0.362. For explainability, SHAP (Shapley Additive Explanations) analysis combined with a five-fold out-of-fold (OOF) summary and beeswarm visualizations was used to examine each item’s contribution to classification. Items 38, 24, 27, and 25 demonstrated the highest discriminative power. The sample’s risk distribution (classes 1–5) was 8.2%, 42.3%, 36.1%, 13.4%, and 0.0%, respectively, with medium + high risk accounting for approximately 49.5%. Within these groups, gender distribution was balanced, education was predominantly associate or undergraduate level, and the 20–30 age range was dominant. Overall, the findings suggest that this unidimensional and internally consistent scale, coupled with a well-calibrated RF model and SHAP-based explanations, provides a robust and practical framework capable of supporting risk profiling, early screening, and targeted intervention planning in both research and applied contexts.

Keywords

• Social media
• Risk profiling
• Exploratory factor analysis
• Machine learning
• Explainable artificial intelligence

Sosyal Medya Risk Düzeyinin Psikometrik ve Açıklanabilir Yapay Zekâ Destekli Makine Öğrenmesi Analizi

Abstract

Bu çalışmanın amacı, sosyal medya kullanımına ilişkin öz-bildirimlere dayalı risk düzeylerini güvenilir ve şeffaf biçimde sınıflandırmaktır. Bu amaç doğrultusunda, beşli risk sınıfı; psikometrik ölçütler, makine öğrenmesi (ML) tabanlı sınıflandırma ve açıklanabilir yapay zekâ (XAI) yaklaşımlarıyla birlikte değerlendirilmiştir. Analiz, Likert türünde ve 1–5 aralığına eşlenmiş maddelere (1=en düşük–5=en yüksek) dayanmaktadır. Psikometrik incelemede keşfedici faktör analizi (EFA) kapsamında Kaiser–Meyer–Olkin (KMO)=0.862, Bartlett küresellik testi (BTS) P<0.001, Cronbach alfa (α)=0.863 ve EV1/EV2=3.71 değerleri, ölçeğin tek boyutlu ve güvenilir olduğunu göstermektedir. Rastgele Orman (RF) ile çok sınıflı tahmin yapılmış, model olasılıkları izotonik kalibrasyon ile düzeltilmiştir; doğrulamada ROC-AUC=0.959, PR-AUC=0.868, Brier=0.255 ve LogLoss=0.362 elde edilmiştir. Açıklanabilirlik için Shapley Katkı Açıklamaları (SHAP) ve 5 kat dış-örnek (OOF) özet ile beeswarm yöntemleri kullanılarak maddelerin sınıflandırmaya katkıları incelenmiş; 38, 24, 27 ve 25 numaralı maddeler en yüksek ayrım gücünü göstermiştir. Örneklemde risk dağılımı (1–5) sırasıyla %8.2, %42.3, %36.1, %13.4, %0.0 olup orta+yüksek risk yaklaşık %49.5’tir. Bu iki grupta cinsiyet dengelidir, eğitim çoğunlukla ön lisans/lisans düzeyindedir ve 20–30 yaş aralığı baskındır. Bulgular, tek boyutlu ve iç tutarlılığı yüksek ölçeğin, iyi kalibre RF modeli ve SHAP tabanlı açıklamalarla birlikte, araştırma ve uygulamada risk profilleme, erken tarama ve hedefli müdahale planlamalarını destekleyebilecek güçlü ve uygulanabilir bir yaklaşım sunduğunu göstermektedir.

Keywords

• Sosyal medya
• Risk profilleme
• Keşfedici faktör analizi
• Makine öğrenimi
• Açıklanabilir yapay zekâ

Supporting Institution

Bu çalışma için herhangi bir kurum ve/veya kuruluştan destek alınmamıştır.

Ethical Statement

Bu araştırma, Hacettepe Üniversitesi Sosyal ve Beşeri Bilimler Araştırma Etik Kurulu’nun 21 Ocak 2025 tarih ve E-66777842-300-00004014095 karar numaralı onayı ile yürütülmüştür. Tüm katılımcılardan araştırmaya gönüllü olarak katılım sağladıklarına ilişkin bilgilendirilmiş onam alınmıştır.

References

1. Andreassen, C. S. (2015). Online social network site addiction: A comprehensive review. Current Addiction Reports, 2(2), 175-184. https://doi.org/10.1007/s40429-015-0056-9
2. Kuss, D. J., & Griffiths, M. D. (2017). Social networking sites and addiction: Ten lessons learned. international Journal of Environmental Research and Public Health, 14(3), 1-17. https://doi.org/doi:10.3390/ijerph14030311
3. Griffiths, M. (2005). A components model of addiction within a biopsychosocial framework. Journal of Substance Use, 10(4), 191-197. https://doi.org/10.1080/14659890500114359
4. Andreassen, C. S., Pallesen, S., & Griffiths, M. D. (2017). The relationship between addictive use of social media, narcissism, and self-esteem: Findings from a large national survey. Addictive Behaviors, 64(2017), 287-293. https://doi.org/https://doi.org/10.1016/j.addbeh.2016.03.006
5. Lin, C.-Y., Broström, A., Nilsen, P., Griffiths, M. D., & Pakpour, A. H. (2017). Psychometric validation of the Persian Bergen Social Media Addiction Scale using classic test theory and Rasch models. Journal of behavioral addictions, 6(4), 620-629. https://doi.org/10.1556/2006.6.2017.071
6. Niculescu-Mizil, A., & Caruana, R. (2005, August 7-11). Predicting good probabilities with supervised learning [Conference presentation]. Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany. https://doi.org/10.1145/1102351.1102430
7. Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. 31st Conference on Neural Information Processing Systems (NIPS 2017). https://arxiv.org/abs/1705.07874
8. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32. https://doi.org/10.1023/A:1010933404324

9. Doğan, A. (2024). Prediction of major earthquakes that may affect İzmir using machine learning methods. Yerbilimleri, 45(2), 93-106. https://doi.org/10.17824/yerbilimleri.1402618
10. Fernández-Delgado, M., Cernadas, E., Barro, S., & Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1), 3133-3181.
11. Biau, G. & Scornet, E. (2016). A random forest guided tour. TEST, 25(2), 197-227. https://doi.org/10.1007/s11749-016-0481-7
12. Cutler, D. R., Edwards Jr, T.C., Beard, K. H., Cutler, A., Hess, K. T., Gibson, J., & Lawler, J. J. (2007). Random forests for classification in ecology. Ecology, 88(11) 2783-2792. https://doi.org/10.1890/07-0539.1
13. Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P. (2023). Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9), 3211-3260. https://doi.org/10.1007/s10994-023-06336-7
14. Vimbi, V., Shaffi, N., & Mahmud, M. (2024). Interpreting artificial intelligence models: a systematic review on the application of LIME and SHAP in Alzheimer’s disease detection. Brain Informatics, 11(1), 10. https://doi.org/10.1186/s40708-024-00222-1
15. Altmann, A., Toloşi, L., Sander, O., & Lengauer, T. (2010). Permutation importance: a corrected feature importance measure. Bioinformatics, 26(10), 1340-1347. https://doi.org/10.1093/bioinformatics/btq134
16. Arévalo-Cordovilla, F. E., & Peña, M. (2025). Evaluating ensemble models for fair and interpretable prediction in higher education using multimodal data. Scientific Reports, 15(29420) 1-14. https://doi.org/10.1038/s41598-025-15388-9
17. Ponce‐Bobadilla, A. V., Schmitt, V., Maier, C. S., Mensing, S., & Stodtmann, S. (2024). Practical guide to SHAP analysis: Explaining supervised machine learning model predictions in drug development. Clinical and Translational Science, 17(e70056), 1-15. https://doi.org/10.1111/cts.70056
18. Zhou, W., Yan, Z., & Zhang, L. (2024). A comparative study of 11 non-linear regression models highlighting autoencoder, DBN, and SVR, enhanced by SHAP importance analysis in soybean branching prediction. Scientific Reports, 14(5905), 1-16. https://doi.org/10.1038/s41598-024-55243-x
19. Doğan, A., & Ünal, C. (2024). Üniversite yerleşme başarısını etkileyen faktörlerin yapay zekâ yöntemleriyle araştırılması. Türk Eğitim Bilimleri Dergisi, 22(3), 1678-1698. https://doi.org/10.37217/tebd.1438947
20. Gan, Y., Kuang, L., Xu, X.-M., Ai, M., He, J.-L., Wang, W., Hong, S., Chen, J.m., Cao, J., & Zhang, Q. (2025). Application of machine learning in predicting adolescent Internet behavioral addiction. Frontiers in Psychiatry, 15(2025), 1-8. https://doi.org/10.3389/fpsyt.2024.1521051
21. Liu, J., Chen, L., Chen, Y., Luo, J., Yu, K., Fan, L., Yong, C., He, H., Liao, S., Ge, Z., & Jiang, L. (2025). Explainable machine learning prediction of internet addiction among Chinese primary and middle school children and adolescents: A longitudinal study based on positive youth development data (2019–2022). Frontiers in Public Health, 13(2025), 1-13. https://doi.org/10.3389/fpubh.2025.1590689
22. Zarate, D., Hobson, B. A., March, E., Griffiths, M. D., & Stavropoulos, V. (2023). Psychometric properties of the Bergen Social Media Addiction Scale: An analysis using item response theory. Addictive Behaviors Reports, 17(100473), 1-8. https://doi.org/https://doi.org/10.1016/j.abrep.2022.100473
23. Yu, D. J., Wing, Y. K., Li, T. M. H., & Chan, N. Y. (2024). The impact of social media use on sleep and mental health in youth: a scoping review. Current Psychiatry Reports, 26(3), 104-119. https://doi.org/10.1007/s11920-024-01481-9
24. Peprah, P., Oduro, M. S., Atta-Osei, G., Addo, I. Y., Morgan, A. K., & Gyasi, R. M. (2024). Problematic social media use mediates the effect of cyberbullying victimisation on psychosomatic complaints in adolescents. Scientific Reports, 14(9773), 1-12. https://doi.org/10.1038/s41598-024-59509-2