A Comparative Analysis of Transformer Architectures for Sentiment and Emotion Classification

Abstract

This paper is a comparison of four Transformer model (BERT, ALBERT, T5, and XLNet) in the case of sentiment analysis of tasks based on data from social media. Used were two data sets: the X (Twitter) data set with tweets of messages related to games and the Emotion data set labeled in the anger, joy, and fear categories. Trained was the model in the same settings of preprocessing, training, as well as in the test settings for 3, 5, 7, and 10 epochs. Presented were results that indicated that the accuracy increased with the higher number of epochs. The maximum accuracies occurred in the case of the BERT model—88.63% in the case of the X data set as well as in the case of the Emotion data set, 97.05%. XLNet established great potential for long-range dependencies, and ALBERT obtained balanced performance due to lightweight architecture. On the contrary, performance of T5 was less in comparison to others. Generally, it could be inferred that Transformer architecture is superior to traditional machine learning technique in the context of sentiment analysis due to higher accuracy and better contextual understanding.

Keywords

• Albert
• Natural Language Processing
• Sentiment Classification
• Transformer Models

Duygu ve Duygusal Durum Sınıflandırması İçin Transformer Mimarilerinin Karşılaştırmalı Analizi

Abstract

Bu makale, sosyal medya verilerine dayalı duygu analizi görevlerinde dört farklı Transformer modelinin (BERT, ALBERT, T5 ve XLNet) karşılaştırılmasını içermektedir. Çalışmada iki veri seti kullanılmıştır: oyunlarla ilgili tweet’leri içeren X (Twitter) veri seti ve öfke, sevinç ve korku kategorilerinde etiketlenmiş Emotion veri seti. Modeller; ön işleme, eğitim ve test ayarlarında aynı koşullarda, 3, 5, 7 ve 10 epoch boyunca eğitilmiştir. Elde edilen sonuçlar, epoch sayısı arttıkça doğruluğun yükseldiğini göstermektedir. En yüksek doğruluk oranları BERT modeli için elde edilmiştir: X veri setinde %88,63, Emotion veri setinde ise %97,05. XLNet uzun menzilli bağımlılıkları yakalama konusunda yüksek potansiyel sergilerken, ALBERT hafif mimarisi sayesinde dengeli bir performans göstermiştir. Buna karşın T5 modeli, diğer modellere kıyasla daha düşük performans sergilemiştir.

Keywords

• Albert
• Doğal dil işleme
• Duygu Sınıflandırması
• Transformer Modelleri

References

1. [1] P. Nandwani and R. Verma, “A review on sentiment analysis and emotion detection from text,” Soc Netw Anal Min, vol. 11, no. 1, p. 81, 2021.
2. [2] S. Ghosh, A. Ekbal, and P. Bhattacharyya, “Natural language processing and sentiment analysis: perspectives from computational intelligence,” in Computational intelligence applications for text and sentiment data analysis, Elsevier, 2023, pp. 17–47.
3. [3] E. Kına, “TRANSFORMER TABANLI DUYGU SINIFLANDIRMASI İLE SOSYAL MEDYADA RUH SAĞLIĞINA İLİŞKİN TÜRKÇE YORUMLARIN ANALİZİ,” Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi, vol. 28, no. 3, pp. 1499–1511, 2025.
4. [4] E. Kına and E. Biçek, “Machine Learning Approach for Emotion Identification and Classification in Bitcoin Sentiment Analysis,” Yüzüncü Yıl Üniversitesi Fen Bilimleri Enstitüsü Dergisi, vol. 29, no. 3, pp. 913–926, 2024, doi: https://doi.org/10.53433/yyufbed.1532649.
5. [5] E. Kına and E. Biçek, “Tweetlerin Duygu Analizi İçin Hibrit Bir Yaklaşım,” Doğu Fen Bilimleri Dergisi, vol. 6, no. 1, pp. 57–68, 2023, doi: 10.57244/DFBD.1314901.
6. [6] H. Bashiri and H. Naderi, “Comprehensive review and comparative analysis of transformer models in sentiment analysis,” Knowl Inf Syst, vol. 66, no. 12, pp. 7305–7361, Dec. 2024, doi: 10.1007/S10115-024-02214-3/FIGURES/10.
7. [7] F. A. Acheampong, H. Nunoo-Mensah, and W. Chen, “Transformer models for text-based emotion detection: a review of BERT-based approaches,” Artif Intell Rev, vol. 54, no. 8, pp. 5789–5829, Dec. 2021, doi: 10.1007/S10462-021-09958-2/TABLES/18.
8. [8] A. Areshey and H. Mathkour, “Exploring transformer models for sentiment classification: A comparison of BERT, RoBERTa, ALBERT, DistilBERT, and XLNet,” Expert Syst, vol. 41, no. 11, p. e13701, Nov. 2024, doi: 10.1111/EXSY.13701.

9. [9] S. Alaparthi and M. Mishra, “Bidirectional Encoder Representations from Transformers (BERT): A sentiment analysis odyssey,” Jul. 2020, Accessed: Oct. 04, 2025. [Online]. Available: https://arxiv.org/pdf/2007.01127
10. [10] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” arXiv preprint arXiv:1909.11942, 2019.
11. [11] M. S. I. Sajol, A. S. M. J. Hasan, M. S. Islam, and M. S. Rahman, “Transforming Social Media Analysis: TweetEval Benchmarking with Advanced Transformer Models,” ISMSIT 2024 - 8th International Symposium on Multidisciplinary Studies and Innovative Technologies, Proceedings, 2024, doi: 10.1109/ISMSIT63511.2024.10757178.
12. [12] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” Adv Neural Inf Process Syst, vol. 32, 2019.
13. [13] E. Kına and E. Biçek, “Duygu Analizinde Denetimli Makine Öğrenme Algoritmalarının Karşılaştırılmaları,(Kahramanmaraş Depremi Örneği),” Batman Üniversitesi Yaşam Bilimleri Dergisi, vol. 13, no. 1, pp. 21–31, 2023.
14. [14] E. Kına and R. Özdağ, “Deep Learning vs. Machine Learning in Sentiment Classification: A Comparative Analysis of Mobile Game Tweets from the X Platform,” Erzincan University Journal of Science and Technology, vol. 18, no. 2, pp. 639–658, 2025.
15. [15] I. D. Hayatu, S. Singh, M. M. Muhammad, R. Mishra, and M. Mishra, “Emotion detection in text data: a comparative study of machine learning algorithms,” Brazilian Journal of Biometrics, vol. 43, no. 4, pp. 1–13, Aug. 2025, doi: 10.28951/BJB.V43I4.786.
16. [16] M. E. Chatzimina, H. A. Papadaki, C. Pontikoglou, and M. Tsiknakis, “A Comparative Sentiment Analysis of Greek Clinical Conversations Using BERT, RoBERTa, GPT-2, and XLNet,” Bioengineering 2024, Vol. 11, Page 521, vol. 11, no. 6, p. 521, May 2024, doi: 10.3390/BIOENGINEERING11060521.
17. [17] Z. Sokolová, M. Harahus, M. Sokol, E. Kupcová, and M. Pleva, “Sentiment Analysis Using Transformer Models: BERT, T5, and GPT,” Proceedings of the International Conference Radioelektronika, RADIOELEKTRONIKA, no. 2025, 2025, doi: 10.1109/RADIOELEKTRONIKA65656.2025.11008427.
18. [18] K. L. Tan, C. P. Lee, K. S. M. Anbananthen, and K. M. Lim, “RoBERTa-LSTM: A Hybrid Model for Sentiment Analysis With Transformer and Recurrent Neural Network,” IEEE Access, vol. 10, pp. 21517–21525, 2022, doi: 10.1109/ACCESS.2022.3152828.
19. [19] A. Branco, D. Parada, M. Silva, F. Mendonça, S. S. Mostafa, and F. Morgado-Dias, “Sentiment Analysis in Portuguese Restaurant Reviews: Application of Transformer Models in Edge Computing,” Electronics 2024, Vol. 13, Page 589, vol. 13, no. 3, p. 589, Jan. 2024, doi: 10.3390/ELECTRONICS13030589.
20. [20] G. Kaur, · Saemundur Haraldsson, and A. Bracciali, “Comparative analysis of transformer models for sentiment classification of UK CBDC discourse on X,” Discover Analytics 2025 3:1, vol. 3, no. 1, pp. 1–39, Jun. 2025, doi: 10.1007/S44257-025-00035-4.
21. [21] S. S. Almalki, “Sentiment Analysis and Emotion Detection Using Transformer Models in Multilingual Social Media Data.,” International Journal of Advanced Computer Science & Applications, vol. 16, no. 3, 2025.
22. [22] K. Taneja, J. Vashishtha, and S. Ratnoo, “Transformer Based Unsupervised Learning Approach for Imbalanced Text Sentiment Analysis of E-Commerce Reviews,” Procedia Comput Sci, vol. 235, pp. 2318–2331, Jan. 2024, doi: 10.1016/J.PROCS.2024.04.220.
23. [23] H. Ali, E. Hashmi, S. Yayilgan Yildirim, and S. Shaikh, “Analyzing Amazon Products Sentiment: A Comparative Study of Machine and Deep Learning, and Transformer-Based Techniques,” Electronics 2024, Vol. 13, Page 1305, vol. 13, no. 7, p. 1305, Mar. 2024, doi: 10.3390/ELECTRONICS13071305.
24. [24] A. Vaswani et al., “Attention is all you need,” Adv Neural Inf Process Syst, vol. 30, no. 1, pp. 5998–6008, 2017.
25. [25] M. N. Razali, N. Arbaiy, P. C. Lin, and S. Ismail, “Optimizing Multiclass Classification Using Convolutional Neural Networks with Class Weights and Early Stopping for Imbalanced Datasets,” Electronics 2025, Vol. 14, Page 705, vol. 14, no. 4, p. 705, Feb. 2025, doi: 10.3390/ELECTRONICS14040705.
26. [26] M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Inf Process Manag, vol. 45, no. 4, pp. 427–437, Jul. 2009, doi: 10.1016/J.IPM.2009.03.002.
27. [27] A. Bello, S. C. Ng, and M. F. Leung, “A BERT Framework to Sentiment Analysis of Tweets,” Sensors 2023, Vol. 23, Page 506, vol. 23, no. 1, p. 506, Jan. 2023, doi: 10.3390/S23010506.
28. [28] T. Bikku, J. Jarugula, L. Kongala, N. D. Tummala, and N. Vardhani Donthiboina, “Exploring the Effectiveness of BERT for Sentiment Analysis on Large-Scale Social Media Data,” 2023 3rd International Conference on Intelligent Technologies, CONIT 2023, 2023, doi: 10.1109/CONIT59222.2023.10205600.
29. [29] S. Pandya, “Comparative Analysis of Large Language Models and Traditional Methods for Sentiment Analysis of Tweets Dataset,” Int. J. Innov. Sci. Res. Technol, vol. 9, no. 12, pp. 1647–1657, 2024.
30. [30] Y. Wang and Y. Wu, “XLNet-LSTM-CNN for text sentiment analysis,” 2024.
31. [31] Z. Ye, T. Zuo, W. Chen, Y. Li, and Z. Lu, “Textual emotion recognition method based on ALBERT-BiLSTM model and SVM-NB classification,” Soft comput, vol. 27, no. 8, pp. 5063–5075, Apr. 2023, doi: 10.1007/S00500-023-07924-4/FIGURES/18.
32. [32] F. Li, J. Li, and F. Abza, “Sentiment analysis of tweets employing convolutional neural network optimized by enhanced gorilla troops optimization algorithm,” Sci Rep, vol. 15, no. 1, pp. 1–20, Dec. 2025, doi: 10.1038/S41598-025-85392-6;SUBJMETA.
33. [33] S. Tam, R. Ben Said, and Ö. Tanriöver, “A ConvBiLSTM Deep Learning Model-Based Approach for Twitter Sentiment Classification,” IEEE Access, vol. 9, pp. 41283–41293, 2021, doi: 10.1109/ACCESS.2021.3064830.
34. [34] S. Riyadi, F. Daffa, C. Damarjati, and M. S. A. M. Ali, “Sentiment Analysis on Social Media Using CNN-RNN Hybrid: A Case Study of Indonesian Presidential Candidate,” Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control, 2025.
35. [35] M. Liebenlito, N. Inayah, E. Choerunnisa, T. E. Sutanto, and S. Inna, “Active learning on Indonesian Twitter sentiment analysis using uncertainty sampling,” Journal of Applied Data Sciences, vol. 5, no. 1, pp. 114–121, Jan. 2024, doi: 10.47738/JADS.V5I1.144.
36. [36] A. B. Alawi and F. Bozkurt, “A hybrid machine learning model for sentiment analysis and satisfaction assessment with Turkish universities using Twitter data,” Decision Analytics Journal, vol. 11, p. 100473, Jun. 2024, doi: 10.1016/J.DAJOUR.2024.100473.