BERT, DistilBERT, RoBERTa ve ELECTRA Doğal Dil İşleme Modelleri ile Çok Sınıflı Haber Sınıflandırması

Year 2026, Volume: 14 Issue: 1, 117 - 129, 21.01.2026

Arafat Şentürk , Ahmet Albayrak , Serdar Arpacı

Abstract

Günümüzde dijital haber kaynaklarının hızla çoğalması, büyük ölçekli metin verilerinin etkin biçimde analiz edilmesini ve sınıflandırılmasını gerekli kılmaktadır. Bu çalışmada, çok sınıflı haber metinlerinin otomatik olarak sınıflandırılması amacıyla BERT (Bidirectional Encoder Representations from Transformers) ve onun türevi olan DistilBERT, RoBERTa ve ELECTRA modelleri karşılaştırmalı olarak değerlendirilmiştir. Her bir model, farklı haber kategorilerine ait metinlerin bağlamsal ve semantik özelliklerini öğrenerek sınıflandırma görevini gerçekleştirmiştir. Modellerin doğruluk, kesinlik, duyarlılık ve F1 skorları gibi çeşitli metrikler üzerinden performansları analiz edilmiştir. DistilBERT modeli, 0,92 doğruluk ve 0,92 ortalama F1 skoru ile en iyi performansı sergilemiştir. Elde edilen bulgular, transformer tabanlı modellerin haber sınıflandırma görevlerinde güçlü bir performans sergilediğini ortaya koymakta; ayrıca model mimarileri arasındaki farkların sınıflandırma başarımı üzerindeki etkisini göstermektedir. Bu sayede, farklı dil modeli mimarilerinin pratik uygulamalarda ne ölçüde etkili olabileceğine dair önemli çıkarımlar elde edilmiştir.

Keywords

BERT , DistilBERT , Doğal Dil İşleme , ELECTRA , Haber Sınıflandırma , Metin Sınıflandırma , RoBERTa

Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models

Abstract

The rapid proliferation of digital news sources today necessitates the effective analysis and classification of large-scale textual data. In this study, BERT (Bidirectional Encoder Representations from Transformers) and its derivatives — DistilBERT, RoBERTa, and ELECTRA — were comparatively evaluated for the automatic classification of multi-class news texts. Each model performed the classification task by learning the contextual and semantic features of texts belonging to different news categories. The models’ performances were analyzed based on various metrics such as accuracy, precision, recall, and F1 score. Among them, the DistilBERT model demonstrated the best performance, achieving an accuracy of 0.92 and a mean F1 score of 0.92. The findings reveal that transformer-based models exhibit strong performance in news classification tasks and further illustrate the impact of architectural differences among these models on classification success. Accordingly, important insights have been gained regarding the practical effectiveness of different language model architectures.

Keywords

BERT , DistilBERT , RoBERTa , ELECTRA , Natural language processing , News classification , Text classification

Ethical Statement

This study does not involve human or animal participants. All procedures followed scientific and ethical principles, and all referenced studies are appropriately cited.

Supporting Institution

This research received no external funding.

Thanks

The authors do not wish to acknowledge any individual or institution.

References

• Anand, S., & Prakasam, P. (2024). Deep learning-based text news classification using bi-directional LSTM model. In 2024 3rd International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024). https://doi.org/10.1109/AIIoT58432.2024.10574679
• Aydın, Ö., & Kantarcı, H. (2024). Türkçe anahtar sözcük çıkarımında LSTM ve BERT tabanlı modellerin karşılaştırılması. Bilgisayar Bilimleri ve Mühendisliği Dergisi, 17(1), 9–18. https://doi.org/10.54525/bbmd.1454220
• Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020). ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv Preprint arXiv:2003.10555. http://arxiv.org/abs/2003.10555
• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171–4186. https://doi.org/10.48550/arXiv.1810.04805
• Dos Santos, D. P., Da Costa, J. P. J., Da Silva, D. A., Mendonca, F., Veiga, C., & De Sousa, R. T. (2023). Multi-class text classification based in oversampling for highly imbalanced dataset. Proceedings of the 22nd IEEE International Conference on Machine Learning and Applications (ICMLA 2023) (pp. 752–755). https://doi.org/10.1109/ICMLA58977.2023.00109
• Dvořáčková, L. (2025). Analyzing word embeddings and their impact on semantic similarity: Through extreme simulated conditions to real dataset characteristics. Neural Computing and Applications, 37(19), 13765–13793. https://doi.org/10.1007/S00521-025-11231-4
• Hassan, S. U., Ahamed, J., & Ahmad, K. (2022). Analytics of machine learning-based algorithms for text classification. Sustainable Operations and Computers, 3, 238–248. https://doi.org/10.1016/j.susoc.2022.03.001
• İzdaş, T., İskifoğlu, H., & Diri, B. (2025). Occupation prediction from twitter data. Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, 27(80), 267–271. https://doi.org/10.21205/deufmd.2025278013
• Joachims, T. (1998). Text categorization with Support Vector Machines: Learning with many relevant features. In C. Nédellec & C. Rouveirol (Eds.), Machine learning: ECML‑98 (Lecture Notes in Computer Science, Vol. 1398, pp. 137–142). Springer. https://doi.org/10.1007/BFb0026683
• Joseph, J., Vineetha, S., & Sobhana, N. V. (2022). A survey on deep learning based sentiment analysis. Materials Today: Proceedings, 58(1), 456–460. https://doi.org/10.1016/j.matpr.2022.02.483
• Joulin, A., Grave, É., Bojanowski, P., & Mikolov, T. (2017). Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2, 427–431. https://aclanthology.org/E17-2068/
• Keswani, A., Jain, T., & Sharma, B. (2023). Multi-class text classification using machine learning & deep learning. In 2023 2nd International Conference on Futuristic Technologies (INCOFT 2023). https://doi.org/10.1109/INCOFT60753.2023.10425423
• Kowsari, K., Meimandi, K. J., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4), Article 150. https://doi.org/10.3390/INFO10040150
• Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR 2020). https://openreview.net/forum?id=H1eA7AEtvS
• Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P. S., & He, L. (2022). A survey on text classification: From traditional to deep learning. ACM Transactions on Intelligent Systems and Technology, 13(2), Article 31. https://doi.org/10.1145/3495162
• Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020). Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2), 261–318. https://doi.org/10.1007/s11263-019-01247-4
• Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv Preprint arXiv:1907.11692. http://arxiv.org/abs/1907.11692
• Mao, K., Xu, J., Yao, X., Qiu, J., Chi, K., & Dai, G. (2022). A text classification model via multi-level semantic features. Symmetry, 14(9), Article 1939. https://doi.org/10.3390/sym14091938
• Mehta, D. K., Patel, M., Dangi, A., Patwa, N., Patel, Z., Jain, R., Shah, P., & Suthar, B. (2024). Exploring the efficacy of natural language processing and supervised learning in the classification of fake news articles. Advances in Robotic Technology, 2(1), 1–6. https://doi.org/10.23880/art-16000108
• Mohammed, A., & Kora, R. (2022). An effective ensemble deep learning framework for text classification. Journal of King Saud University - Computer and Information Sciences, 34(10), 8825–8837. https://doi.org/10.1016/j.jksuci.2021.11.001
• Nagendra, N., & Chandra, J. (2024). Hybrid approach for multi-classification of news documents using artificial intelligence. In Proceedings of the 2024 5th International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV 2024) (pp. 466–473). https://doi.org/10.1109/ICICV62344.2024.00079
• Ni, S., & Kao, H.-Y. (2022). ELECTRA is a zero-shot learner, too. arXiv Preprint arXiv:2207.08141. https://doi.org/10.48550/arXiv.2207.08141
• Pancholi, S., & Joshi, A. M. (2022). Advanced energy kernel-based feature extraction scheme for improved EMG-PR-based prosthesis control against force variation. IEEE Transactions on Cybernetics, 52(5), 3819–3828. https://doi.org/10.1109/TCYB.2020.3016595
• Peng, J., & Han, K. (2021). Survey of pre-trained models for natural language processing. In 2021 IEEE International Conference on Electronic Communications, Internet of Things and Big Data (ICEIB 2021) (pp. 277–280). https://doi.org/10.1109/ICEIB53692.2021.9686420
• Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, 842–866. https://doi.org/10.1162/TACL_A_00349
• Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv Preprint arXiv:1910.01108. https://doi.org/10.48550/arXiv.1910.01108
• Santosh Kumar, P., Yadav, R. B., & Dhavale, S. V. (2021). A comparison of pre-trained word embeddings for sentiment analysis using deep learning. In D. Gupta, A. Khanna, S. Bhattacharyya, A. E. Hassanien, S. Anand, & A. Jaiswal (Eds.), International conference on innovative computing and communications (Advances in Intelligent Systems and Computing, Vol. 1165, pp. 525–537). Springer. https://doi.org/10.1007/978-981-15-5113-0_41
• Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1–47. https://doi.org/10.1145/505282.505283
• Sridhar, S., Rethinapandian, Y., Kushala, T. K., Kritiish, M., & Chowdary, P. K. (2022). A novel approach towards news category prediction using NLP. In International Conference on Edge Computing and Applications (ICECAA 2022) (pp. 244–247). https://doi.org/10.1109/ICECAA55415.2022.9936571
• Tan, M., & Bakır, H. (2025). Fake news detection using BERT and Bi-LSTM with grid search hyperparameter optimization. Bilişim Teknolojileri Dergisi, 18(1), 11–28. https://doi.org/10.17671/gazibtd.1521520
• Vargas, V. M., Guijo-Rubio, D., Gutiérrez, P. A., & Hervás-Martínez, C. (2021). ReLU-based activations: Analysis and experimental study for deep learning. In I. Rojas, G. Joya, & A. Catalá (Eds.), Advances in artificial intelligence (Lecture Notes in Computer Science, Vol. 12882, pp. 33–43). Springer. https://doi.org/10.1007/978-3-030-85713-4_4
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. arXiv Preprint arXiv:1706.03762. https://doi.org/10.48550/arXiv.1706.03762
• Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019). XLNet: Generalized autoregressive pretraining for language understanding. arXiv Preprint arXiv:1906.08237. https://doi.org/10.48550/arXiv.1906.08237
• Yücesoy Kahraman, S., Durmuşoğlu, A., & Dereli, T. (2024). Ön eğitimli Bert modeli ile patent sınıflandırılması. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 39(4), 2484–2496. https://doi.org/10.17341/gazimmfd.1292543
• Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., & Ahmed, A. (2020). Big Bird: Transformers for longer sequences. arXiv Preprint arXiv:2007.14062. https://doi.org/10.48550/arXiv.2007.14062
• Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4), Article e1253. https://doi.org/10.1002/widm.1253
• Zhang, X., Zhao, J., & Lecun, Y. (2016). Character-level convolutional networks for text classification. arXiv Preprint arXiv:1509.01626. https://doi.org/10.48550/arXiv.1509.01626
• Zhang, Z., Das, A., Rahgouy, M., Bao, Y., & Baskiyar, S. (2023). Multi-label classification of cs papers using natural language processing models. Proceedings of the 22nd IEEE International Conference on Machine Learning and Applications (ICMLA 2023) (pp. 1907–1912). https://doi.org/10.1109/ICMLA58977.2023.00289