Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models

Arafat Şentürk; Ahmet Albayrak; Serdar Arpacı

doi:10.29130/dubited.1737003

TR EN

BERT, DistilBERT, RoBERTa ve ELECTRA Doğal Dil İşleme Modelleri ile Çok Sınıflı Haber Sınıflandırması

Abstract

Günümüzde dijital haber kaynaklarının hızla çoğalması, büyük ölçekli metin verilerinin etkin biçimde analiz edilmesini ve sınıflandırılmasını gerekli kılmaktadır. Bu çalışmada, çok sınıflı haber metinlerinin otomatik olarak sınıflandırılması amacıyla BERT (Bidirectional Encoder Representations from Transformers) ve onun türevi olan DistilBERT, RoBERTa ve ELECTRA modelleri karşılaştırmalı olarak değerlendirilmiştir. Her bir model, farklı haber kategorilerine ait metinlerin bağlamsal ve semantik özelliklerini öğrenerek sınıflandırma görevini gerçekleştirmiştir. Modellerin doğruluk, kesinlik, duyarlılık ve F1 skorları gibi çeşitli metrikler üzerinden performansları analiz edilmiştir. DistilBERT modeli, 0,92 doğruluk ve 0,92 ortalama F1 skoru ile en iyi performansı sergilemiştir. Elde edilen bulgular, transformer tabanlı modellerin haber sınıflandırma görevlerinde güçlü bir performans sergilediğini ortaya koymakta; ayrıca model mimarileri arasındaki farkların sınıflandırma başarımı üzerindeki etkisini göstermektedir. Bu sayede, farklı dil modeli mimarilerinin pratik uygulamalarda ne ölçüde etkili olabileceğine dair önemli çıkarımlar elde edilmiştir.

Keywords

BERT, DistilBERT, Doğal Dil İşleme, ELECTRA, Haber Sınıflandırma, Metin Sınıflandırma, RoBERTa

Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models

Abstract

The rapid proliferation of digital news sources today necessitates the effective analysis and classification of large-scale textual data. In this study, BERT (Bidirectional Encoder Representations from Transformers) and its derivatives — DistilBERT, RoBERTa, and ELECTRA — were comparatively evaluated for the automatic classification of multi-class news texts. Each model performed the classification task by learning the contextual and semantic features of texts belonging to different news categories. The models’ performances were analyzed based on various metrics such as accuracy, precision, recall, and F1 score. Among them, the DistilBERT model demonstrated the best performance, achieving an accuracy of 0.92 and a mean F1 score of 0.92. The findings reveal that transformer-based models exhibit strong performance in news classification tasks and further illustrate the impact of architectural differences among these models on classification success. Accordingly, important insights have been gained regarding the practical effectiveness of different language model architectures.

Keywords

BERT, DistilBERT, RoBERTa, ELECTRA, Natural language processing, News classification, Text classification

Supporting Institution

This research received no external funding.

Ethical Statement

This study does not involve human or animal participants. All procedures followed scientific and ethical principles, and all referenced studies are appropriately cited.

Thanks

The authors do not wish to acknowledge any individual or institution.

References

Anand, S., & Prakasam, P. (2024). Deep learning-based text news classification using bi-directional LSTM model. In 2024 3rd International Conference on Artificial Intelligence for Internet of Things (AIIoT 2024). https://doi.org/10.1109/AIIoT58432.2024.10574679
Aydın, Ö., & Kantarcı, H. (2024). Türkçe anahtar sözcük çıkarımında LSTM ve BERT tabanlı modellerin karşılaştırılması. Bilgisayar Bilimleri ve Mühendisliği Dergisi, 17(1), 9–18. https://doi.org/10.54525/bbmd.1454220
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020). ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv Preprint arXiv:2003.10555. http://arxiv.org/abs/2003.10555
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171–4186. https://doi.org/10.48550/arXiv.1810.04805
Dos Santos, D. P., Da Costa, J. P. J., Da Silva, D. A., Mendonca, F., Veiga, C., & De Sousa, R. T. (2023). Multi-class text classification based in oversampling for highly imbalanced dataset. Proceedings of the 22nd IEEE International Conference on Machine Learning and Applications (ICMLA 2023) (pp. 752–755). https://doi.org/10.1109/ICMLA58977.2023.00109
Dvořáčková, L. (2025). Analyzing word embeddings and their impact on semantic similarity: Through extreme simulated conditions to real dataset characteristics. Neural Computing and Applications, 37(19), 13765–13793. https://doi.org/10.1007/S00521-025-11231-4
Hassan, S. U., Ahamed, J., & Ahmad, K. (2022). Analytics of machine learning-based algorithms for text classification. Sustainable Operations and Computers, 3, 238–248. https://doi.org/10.1016/j.susoc.2022.03.001
İzdaş, T., İskifoğlu, H., & Diri, B. (2025). Occupation prediction from twitter data. Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, 27(80), 267–271. https://doi.org/10.21205/deufmd.2025278013
Joachims, T. (1998). Text categorization with Support Vector Machines: Learning with many relevant features. In C. Nédellec & C. Rouveirol (Eds.), Machine learning: ECML‑98 (Lecture Notes in Computer Science, Vol. 1398, pp. 137–142). Springer. https://doi.org/10.1007/BFb0026683
Joseph, J., Vineetha, S., & Sobhana, N. V. (2022). A survey on deep learning based sentiment analysis. Materials Today: Proceedings, 58(1), 456–460. https://doi.org/10.1016/j.matpr.2022.02.483

Joulin, A., Grave, É., Bojanowski, P., & Mikolov, T. (2017). Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2, 427–431. https://aclanthology.org/E17-2068/
Keswani, A., Jain, T., & Sharma, B. (2023). Multi-class text classification using machine learning & deep learning. In 2023 2nd International Conference on Futuristic Technologies (INCOFT 2023). https://doi.org/10.1109/INCOFT60753.2023.10425423
Kowsari, K., Meimandi, K. J., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4), Article 150. https://doi.org/10.3390/INFO10040150
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR 2020). https://openreview.net/forum?id=H1eA7AEtvS
Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P. S., & He, L. (2022). A survey on text classification: From traditional to deep learning. ACM Transactions on Intelligent Systems and Technology, 13(2), Article 31. https://doi.org/10.1145/3495162
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020). Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2), 261–318. https://doi.org/10.1007/s11263-019-01247-4
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv Preprint arXiv:1907.11692. http://arxiv.org/abs/1907.11692
Mao, K., Xu, J., Yao, X., Qiu, J., Chi, K., & Dai, G. (2022). A text classification model via multi-level semantic features. Symmetry, 14(9), Article 1939. https://doi.org/10.3390/sym14091938
Mehta, D. K., Patel, M., Dangi, A., Patwa, N., Patel, Z., Jain, R., Shah, P., & Suthar, B. (2024). Exploring the efficacy of natural language processing and supervised learning in the classification of fake news articles. Advances in Robotic Technology, 2(1), 1–6. https://doi.org/10.23880/art-16000108
Mohammed, A., & Kora, R. (2022). An effective ensemble deep learning framework for text classification. Journal of King Saud University - Computer and Information Sciences, 34(10), 8825–8837. https://doi.org/10.1016/j.jksuci.2021.11.001
Nagendra, N., & Chandra, J. (2024). Hybrid approach for multi-classification of news documents using artificial intelligence. In Proceedings of the 2024 5th International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV 2024) (pp. 466–473). https://doi.org/10.1109/ICICV62344.2024.00079
Ni, S., & Kao, H.-Y. (2022). ELECTRA is a zero-shot learner, too. arXiv Preprint arXiv:2207.08141. https://doi.org/10.48550/arXiv.2207.08141
Pancholi, S., & Joshi, A. M. (2022). Advanced energy kernel-based feature extraction scheme for improved EMG-PR-based prosthesis control against force variation. IEEE Transactions on Cybernetics, 52(5), 3819–3828. https://doi.org/10.1109/TCYB.2020.3016595
Peng, J., & Han, K. (2021). Survey of pre-trained models for natural language processing. In 2021 IEEE International Conference on Electronic Communications, Internet of Things and Big Data (ICEIB 2021) (pp. 277–280). https://doi.org/10.1109/ICEIB53692.2021.9686420
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, 842–866. https://doi.org/10.1162/TACL_A_00349
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv Preprint arXiv:1910.01108. https://doi.org/10.48550/arXiv.1910.01108
Santosh Kumar, P., Yadav, R. B., & Dhavale, S. V. (2021). A comparison of pre-trained word embeddings for sentiment analysis using deep learning. In D. Gupta, A. Khanna, S. Bhattacharyya, A. E. Hassanien, S. Anand, & A. Jaiswal (Eds.), International conference on innovative computing and communications (Advances in Intelligent Systems and Computing, Vol. 1165, pp. 525–537). Springer. https://doi.org/10.1007/978-981-15-5113-0_41
Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1–47. https://doi.org/10.1145/505282.505283
Sridhar, S., Rethinapandian, Y., Kushala, T. K., Kritiish, M., & Chowdary, P. K. (2022). A novel approach towards news category prediction using NLP. In International Conference on Edge Computing and Applications (ICECAA 2022) (pp. 244–247). https://doi.org/10.1109/ICECAA55415.2022.9936571
Tan, M., & Bakır, H. (2025). Fake news detection using BERT and Bi-LSTM with grid search hyperparameter optimization. Bilişim Teknolojileri Dergisi, 18(1), 11–28. https://doi.org/10.17671/gazibtd.1521520
Vargas, V. M., Guijo-Rubio, D., Gutiérrez, P. A., & Hervás-Martínez, C. (2021). ReLU-based activations: Analysis and experimental study for deep learning. In I. Rojas, G. Joya, & A. Catalá (Eds.), Advances in artificial intelligence (Lecture Notes in Computer Science, Vol. 12882, pp. 33–43). Springer. https://doi.org/10.1007/978-3-030-85713-4_4
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. arXiv Preprint arXiv:1706.03762. https://doi.org/10.48550/arXiv.1706.03762
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019). XLNet: Generalized autoregressive pretraining for language understanding. arXiv Preprint arXiv:1906.08237. https://doi.org/10.48550/arXiv.1906.08237
Yücesoy Kahraman, S., Durmuşoğlu, A., & Dereli, T. (2024). Ön eğitimli Bert modeli ile patent sınıflandırılması. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 39(4), 2484–2496. https://doi.org/10.17341/gazimmfd.1292543
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., & Ahmed, A. (2020). Big Bird: Transformers for longer sequences. arXiv Preprint arXiv:2007.14062. https://doi.org/10.48550/arXiv.2007.14062
Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4), Article e1253. https://doi.org/10.1002/widm.1253
Zhang, X., Zhao, J., & Lecun, Y. (2016). Character-level convolutional networks for text classification. arXiv Preprint arXiv:1509.01626. https://doi.org/10.48550/arXiv.1509.01626
Zhang, Z., Das, A., Rahgouy, M., Bao, Y., & Baskiyar, S. (2023). Multi-label classification of cs papers using natural language processing models. Proceedings of the 22nd IEEE International Conference on Machine Learning and Applications (ICMLA 2023) (pp. 1907–1912). https://doi.org/10.1109/ICMLA58977.2023.00289

Details

Primary Language

English

Subjects

Deep Learning, Classification Algorithms

Journal Section

Research Article

Authors

Arafat Şentürk ^*
0000-0002-9005-3565
Türkiye

Ahmet Albayrak
0000-0002-2166-1102
Türkiye

Serdar Arpacı
0000-0002-7141-0539
Türkiye

Publication Date

January 21, 2026

Submission Date

July 7, 2025

Acceptance Date

November 10, 2025

Published in Issue

Year 2026 Volume: 14 Number: 1

DOI

https://doi.org/10.29130/dubited.1737003

IZ

https://izlik.org/JA34PS99KZ

APA

Şentürk, A., Albayrak, A., & Arpacı, S. (2026). Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models. Duzce University Journal of Science and Technology, 14(1), 117-129. https://doi.org/10.29130/dubited.1737003

AMA

1.Şentürk A, Albayrak A, Arpacı S. Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models. DUBİTED. 2026;14(1):117-129. doi:10.29130/dubited.1737003

Chicago

Şentürk, Arafat, Ahmet Albayrak, and Serdar Arpacı. 2026. “Multi-Class News Classification With BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models”. Duzce University Journal of Science and Technology 14 (1): 117-29. https://doi.org/10.29130/dubited.1737003.

EndNote

Şentürk A, Albayrak A, Arpacı S (January 1, 2026) Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models. Duzce University Journal of Science and Technology 14 1 117–129.

IEEE

[1]A. Şentürk, A. Albayrak, and S. Arpacı, “Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models”, DUBİTED, vol. 14, no. 1, pp. 117–129, Jan. 2026, doi: 10.29130/dubited.1737003.

ISNAD

Şentürk, Arafat - Albayrak, Ahmet - Arpacı, Serdar. “Multi-Class News Classification With BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models”. Duzce University Journal of Science and Technology 14/1 (January 1, 2026): 117-129. https://doi.org/10.29130/dubited.1737003.

JAMA

1.Şentürk A, Albayrak A, Arpacı S. Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models. DUBİTED. 2026;14:117–129.

MLA

Şentürk, Arafat, et al. “Multi-Class News Classification With BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models”. Duzce University Journal of Science and Technology, vol. 14, no. 1, Jan. 2026, pp. 117-29, doi:10.29130/dubited.1737003.

Vancouver

1.Arafat Şentürk, Ahmet Albayrak, Serdar Arpacı. Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models. DUBİTED. 2026 Jan. 1;14(1):117-29. doi:10.29130/dubited.1737003

BERT, DistilBERT, RoBERTa ve ELECTRA Doğal Dil İşleme Modelleri ile Çok Sınıflı Haber Sınıflandırması

Abstract

Keywords

Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models

Abstract

Keywords

Supporting Institution

Ethical Statement

Thanks

References

Details

Primary Language

Subjects

Journal Section

Authors

Publication Date

Submission Date

Acceptance Date

Published in Issue

DOI

IZ

Cite