BERT, DistilBERT, RoBERTa ve ELECTRA Doğal Dil İşleme Modelleri ile Çok Sınıflı Haber Sınıflandırması

Abstract

Günümüzde dijital haber kaynaklarının hızla çoğalması, büyük ölçekli metin verilerinin etkin biçimde analiz edilmesini ve sınıflandırılmasını gerekli kılmaktadır. Bu çalışmada, çok sınıflı haber metinlerinin otomatik olarak sınıflandırılması amacıyla BERT (Bidirectional Encoder Representations from Transformers) ve onun türevi olan DistilBERT, RoBERTa ve ELECTRA modelleri karşılaştırmalı olarak değerlendirilmiştir. Her bir model, farklı haber kategorilerine ait metinlerin bağlamsal ve semantik özelliklerini öğrenerek sınıflandırma görevini gerçekleştirmiştir. Modellerin doğruluk, kesinlik, duyarlılık ve F1 skorları gibi çeşitli metrikler üzerinden performansları analiz edilmiştir. DistilBERT modeli, 0,92 doğruluk ve 0,92 ortalama F1 skoru ile en iyi performansı sergilemiştir. Elde edilen bulgular, transformer tabanlı modellerin haber sınıflandırma görevlerinde güçlü bir performans sergilediğini ortaya koymakta; ayrıca model mimarileri arasındaki farkların sınıflandırma başarımı üzerindeki etkisini göstermektedir. Bu sayede, farklı dil modeli mimarilerinin pratik uygulamalarda ne ölçüde etkili olabileceğine dair önemli çıkarımlar elde edilmiştir.

Keywords

• BERT
• DistilBERT
• Doğal Dil İşleme
• ELECTRA
• Haber Sınıflandırma
• Metin Sınıflandırma
• RoBERTa

Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models

Abstract

The rapid proliferation of digital news sources today necessitates the effective analysis and classification of large-scale textual data. In this study, BERT (Bidirectional Encoder Representations from Transformers) and its derivatives — DistilBERT, RoBERTa, and ELECTRA — were comparatively evaluated for the automatic classification of multi-class news texts. Each model performed the classification task by learning the contextual and semantic features of texts belonging to different news categories. The models’ performances were analyzed based on various metrics such as accuracy, precision, recall, and F1 score. Among them, the DistilBERT model demonstrated the best performance, achieving an accuracy of 0.92 and a mean F1 score of 0.92. The findings reveal that transformer-based models exhibit strong performance in news classification tasks and further illustrate the impact of architectural differences among these models on classification success. Accordingly, important insights have been gained regarding the practical effectiveness of different language model architectures.

Keywords

• BERT
• DistilBERT
• RoBERTa
• ELECTRA
• Natural language processing
• News classification
• Text classification

References

1. Anand, S., & Prakasam, P. (2024). Deep learning-based text news classification using bi-directional LSTM model. In 2024 3rd International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024). https://doi.org/10.1109/AIIoT58432.2024.10574679
2. Aydın, Ö., & Kantarcı, H. (2024). Türkçe anahtar sözcük çıkarımında LSTM ve BERT tabanlı modellerin karşılaştırılması. Bilgisayar Bilimleri ve Mühendisliği Dergisi, 17(1), 9–18. https://doi.org/10.54525/bbmd.1454220
3. Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020). ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv Preprint arXiv:2003.10555. http://arxiv.org/abs/2003.10555
4. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171–4186. https://doi.org/10.48550/arXiv.1810.04805
5. Dos Santos, D. P., Da Costa, J. P. J., Da Silva, D. A., Mendonca, F., Veiga, C., & De Sousa, R. T. (2023). Multi-class text classification based in oversampling for highly imbalanced dataset. Proceedings of the 22nd IEEE International Conference on Machine Learning and Applications (ICMLA 2023) (pp. 752–755). https://doi.org/10.1109/ICMLA58977.2023.00109
6. Dvořáčková, L. (2025). Analyzing word embeddings and their impact on semantic similarity: Through extreme simulated conditions to real dataset characteristics. Neural Computing and Applications, 37(19), 13765–13793. https://doi.org/10.1007/S00521-025-11231-4
7. Hassan, S. U., Ahamed, J., & Ahmad, K. (2022). Analytics of machine learning-based algorithms for text classification. Sustainable Operations and Computers, 3, 238–248. https://doi.org/10.1016/j.susoc.2022.03.001
8. İzdaş, T., İskifoğlu, H., & Diri, B. (2025). Occupation prediction from twitter data. Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, 27(80), 267–271. https://doi.org/10.21205/deufmd.2025278013

9. Joachims, T. (1998). Text categorization with Support Vector Machines: Learning with many relevant features. In C. Nédellec & C. Rouveirol (Eds.), Machine learning: ECML‑98 (Lecture Notes in Computer Science, Vol. 1398, pp. 137–142). Springer. https://doi.org/10.1007/BFb0026683
10. Joseph, J., Vineetha, S., & Sobhana, N. V. (2022). A survey on deep learning based sentiment analysis. Materials Today: Proceedings, 58(1), 456–460. https://doi.org/10.1016/j.matpr.2022.02.483
11. Joulin, A., Grave, É., Bojanowski, P., & Mikolov, T. (2017). Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2, 427–431. https://aclanthology.org/E17-2068/
12. Keswani, A., Jain, T., & Sharma, B. (2023). Multi-class text classification using machine learning & deep learning. In 2023 2nd International Conference on Futuristic Technologies (INCOFT 2023). https://doi.org/10.1109/INCOFT60753.2023.10425423
13. Kowsari, K., Meimandi, K. J., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4), Article 150. https://doi.org/10.3390/INFO10040150
14. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR 2020). https://openreview.net/forum?id=H1eA7AEtvS
15. Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P. S., & He, L. (2022). A survey on text classification: From traditional to deep learning. ACM Transactions on Intelligent Systems and Technology, 13(2), Article 31. https://doi.org/10.1145/3495162
16. Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020). Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2), 261–318. https://doi.org/10.1007/s11263-019-01247-4
17. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv Preprint arXiv:1907.11692. http://arxiv.org/abs/1907.11692
18. Mao, K., Xu, J., Yao, X., Qiu, J., Chi, K., & Dai, G. (2022). A text classification model via multi-level semantic features. Symmetry, 14(9), Article 1939. https://doi.org/10.3390/sym14091938
19. Mehta, D. K., Patel, M., Dangi, A., Patwa, N., Patel, Z., Jain, R., Shah, P., & Suthar, B. (2024). Exploring the efficacy of natural language processing and supervised learning in the classification of fake news articles. Advances in Robotic Technology, 2(1), 1–6. https://doi.org/10.23880/art-16000108
20. Mohammed, A., & Kora, R. (2022). An effective ensemble deep learning framework for text classification. Journal of King Saud University - Computer and Information Sciences, 34(10), 8825–8837. https://doi.org/10.1016/j.jksuci.2021.11.001
21. Nagendra, N., & Chandra, J. (2024). Hybrid approach for multi-classification of news documents using artificial intelligence. In Proceedings of the 2024 5th International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV 2024) (pp. 466–473). https://doi.org/10.1109/ICICV62344.2024.00079
22. Ni, S., & Kao, H.-Y. (2022). ELECTRA is a zero-shot learner, too. arXiv Preprint arXiv:2207.08141. https://doi.org/10.48550/arXiv.2207.08141
23. Pancholi, S., & Joshi, A. M. (2022). Advanced energy kernel-based feature extraction scheme for improved EMG-PR-based prosthesis control against force variation. IEEE Transactions on Cybernetics, 52(5), 3819–3828. https://doi.org/10.1109/TCYB.2020.3016595
24. Peng, J., & Han, K. (2021). Survey of pre-trained models for natural language processing. In 2021 IEEE International Conference on Electronic Communications, Internet of Things and Big Data (ICEIB 2021) (pp. 277–280). https://doi.org/10.1109/ICEIB53692.2021.9686420
25. Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, 842–866. https://doi.org/10.1162/TACL_A_00349
26. Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv Preprint arXiv:1910.01108. https://doi.org/10.48550/arXiv.1910.01108
27. Santosh Kumar, P., Yadav, R. B., & Dhavale, S. V. (2021). A comparison of pre-trained word embeddings for sentiment analysis using deep learning. In D. Gupta, A. Khanna, S. Bhattacharyya, A. E. Hassanien, S. Anand, & A. Jaiswal (Eds.), International conference on innovative computing and communications (Advances in Intelligent Systems and Computing, Vol. 1165, pp. 525–537). Springer. https://doi.org/10.1007/978-981-15-5113-0_41
28. Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1–47. https://doi.org/10.1145/505282.505283
29. Sridhar, S., Rethinapandian, Y., Kushala, T. K., Kritiish, M., & Chowdary, P. K. (2022). A novel approach towards news category prediction using NLP. In International Conference on Edge Computing and Applications (ICECAA 2022) (pp. 244–247). https://doi.org/10.1109/ICECAA55415.2022.9936571
30. Tan, M., & Bakır, H. (2025). Fake news detection using BERT and Bi-LSTM with grid search hyperparameter optimization. Bilişim Teknolojileri Dergisi, 18(1), 11–28. https://doi.org/10.17671/gazibtd.1521520
31. Vargas, V. M., Guijo-Rubio, D., Gutiérrez, P. A., & Hervás-Martínez, C. (2021). ReLU-based activations: Analysis and experimental study for deep learning. In I. Rojas, G. Joya, & A. Catalá (Eds.), Advances in artificial intelligence (Lecture Notes in Computer Science, Vol. 12882, pp. 33–43). Springer. https://doi.org/10.1007/978-3-030-85713-4_4
32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. arXiv Preprint arXiv:1706.03762. https://doi.org/10.48550/arXiv.1706.03762
33. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019). XLNet: Generalized autoregressive pretraining for language understanding. arXiv Preprint arXiv:1906.08237. https://doi.org/10.48550/arXiv.1906.08237
34. Yücesoy Kahraman, S., Durmuşoğlu, A., & Dereli, T. (2024). Ön eğitimli Bert modeli ile patent sınıflandırılması. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 39(4), 2484–2496. https://doi.org/10.17341/gazimmfd.1292543
35. Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., & Ahmed, A. (2020). Big Bird: Transformers for longer sequences. arXiv Preprint arXiv:2007.14062. https://doi.org/10.48550/arXiv.2007.14062
36. Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4), Article e1253. https://doi.org/10.1002/widm.1253
37. Zhang, X., Zhao, J., & Lecun, Y. (2016). Character-level convolutional networks for text classification. arXiv Preprint arXiv:1509.01626. https://doi.org/10.48550/arXiv.1509.01626
38. Zhang, Z., Das, A., Rahgouy, M., Bao, Y., & Baskiyar, S. (2023). Multi-label classification of cs papers using natural language processing models. Proceedings of the 22nd IEEE International Conference on Machine Learning and Applications (ICMLA 2023) (pp. 1907–1912). https://doi.org/10.1109/ICMLA58977.2023.00289