Sentiment Analysis on GPT-4 with Comparative Models Using Twitter Data

Abstract

Every day, people from all over the world use Twitter to talk about many different topics using hashtags. Since ChatGPT was launched, researchers have been studying how people perceive it in society. This research aims to find out what Turkish Twitter users think about OpenAI’s latest AI model called Generative Pre-trained Transformer 4 (GPT-4). The quantitative data used in this study consist of hashtags on the topic of GPT-4 and involve 2,978 tweets on this topic that were shared on Twitter between March 14-April 9, 2023. The study uses TextBlob sentiment scores to classify the tweets and support vector machines, logistic regression, XGBoost, and random forest algorithms to classify the sentiment of the dataset. The results from the logistic regression, XGBoost, and support vector methods are in close alignment. All parameter findings indicate dependable machine learning, emphasizing the models’ success in classifying tweet sentiment.

Keywords

• Sentiment analysis
• social media
• Twitter
• natural language processing

References

1. Abdullah M., Madain A., & Jararweh Y. (2022). ChatGPT: fundamentals, applications and social impacts. 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS). 1-8. http://dx.doi.org/10.1109/SNAMS58071.2022.10062688 google scholar
2. Ağralı, Ö., & Aydın, Ö. (2021). Tweet classification and sentiment analysis on metaverse related messages,. Journal of Metaverse, 1(1), 25-30. google scholar
3. Aydın, C. (2018). makine öğrenmesi algoritmalari kullanilarak itfaiye istasyonu ihtiyacinin siniflandirilmasi. Avrupa Bilim ve Teknoloji Dergisi, (14), 169-175. google scholar
4. Aydın, Ö., & Karaarslan, E. (2023). Is ChatGPT leading generative AI? What is beyond expectations?. 1-23. Social Science Research Network (SSRN): https://ssrn.com/abstract=4341500 or http://dx.doi.org/10.2139/ssrn.4341500. google scholar
5. Ayhan, S., & Erdoğmuş, Ş. (2014). Destek vektör makineleriyle siniflandirma problemlerinin çözümü için çekirdek fonksiyonu seçimi. Eskişehir Osmangazi Üniversitesi İktisadi ve İdari Bilim Dergisi, 9(1), 175-198. google scholar
6. Agarwal, B., Mittal, N., Bansal, P., & Garg, S. (2015). Sentiment analysis using common-sense and context information. Computational İntelligence and Neuroscience, 2015, 1-9. http://dx.doi.org/10.1155/2015/715730 google scholar
7. Bayram, İ., & Turan, A. (2022). Türkiye’de kripto para farkindaliği ve tutumu: duygu analizi ve istatistiksel analiz ile bir değerlendirme. Yönetim Bilişim Sistemleri Dergisi, 8(2), 20-35. https://dergipark.org.tr/en/pub/ybs/issue/75943/1197985 google scholar
8. Bengesi S., Oladunni T., Olusegun R., & Audu H. (2023). A Machine Learning-Sentiment Analysis on Monkeypox Outbreak: An Extensive Dataset to Show the Polarity of Public Opinion From Twitter Tweets, IEEE Access, 11( 11811-11826). http://dx.doi.org/10.1109/ACCESS.2023.3242290 google scholar

9. Bircan, H. (2004). Lojistik regresyon analizi: Tıp verileri üzerine bir uygulama. Kocaeli Üniversitesi Sosyal Bilimler Dergisi, (8), 185-208. google scholar
10. Biltawi, M., Etaiwi, W., Tedmori, S., Hudaib, A., & Awajan, A. (2016). Sentiment classification techniques for arabic language: A survey. 7th International Conference on Information and Communication Systems (ICICS), 339-346. google scholar
11. Botchu, R., & Iyengar, K. P. (2023). Will ChatGPT drive radiology in the future?. Indian Journal of Radiology and Imaging. https://doi.org/10.1055/s-0043-1769591 google scholar
12. Bruns, A., & Burgess, J. E. (2011). The use of Twitter hashtags in the formation of ad hoc publics. Paper presented at the 6th European Consortium for Political Research General Conference, 25 - 27 August 2011, University of Iceland, Reykjavik. http://eprints.qut.edu.au/46515/ google scholar
13. Dandekar, A. R., Shrotri, A.P., Hargude, N.V., Awate, P.P., & Patil, N.D. (2018,August). The rise of social media and its impact. Internatıonal Journal of Research And Analytıcal Revıews 5(3), 882-886. google scholar
14. Diyasa, I. G. S. M., Mandenni, N. M. I. M., Fachrurrozi, M. I., Pradika, S. I., Manab, K. R. N., & Sasmita, N. R. (2021). Twitter sentiment analysis as an evaluation and service base on python textblob. In IOP Conference Series: Materials Science and Engineering, 1125(1), 1-12. https://doi.org/10.1088/1757-899X/1125/1/012034 google scholar
15. Doğan, G. (2022). Makine öğrenmesi algoritmalari ile betonarme kirişlerin burulma momenti tahmini. El-Cezeri El-Cezerî Fen ve Mühendislik Dergisi, 9(2), 912-924. google scholar
16. Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023). Investigating code generation performance of chatgpt with crowdsourcing social data. In Proceedings of the 47th IEEE Computer Software and Applications Conference, 1-10. google scholar
17. Hardeniya, T., & Borikar, D. A. (2016). Dictionary based approach to sentiment analysis-a review. International Journal of Advanced Engineering, Management and Science, 2(5), 317-322. google scholar
18. Kahya, A. N. (2021). Wikipedia’daki verilere metin madenciliği yöntemlerinin uygulanmasi. Eskişehir Türk Dünyası Uygulama ve Araştırma Merkezi (ESTUDAM) Bilişim Dergisi, 2(1), 11-14. google scholar
19. Kaur C., & Sharma A. (2022). Social issues sentiment analysis using python. 2020 5th International Conference on Computing, Communication and Security (ICCCS), 1-6. http://dx.doi.org/10.1109/ICCCS49678.2020.9277251 google scholar
20. Keskin, E. K. (2023). Yapay zekâ sohbet robotu ChatGPT ve Türkiye internet gündeminde oluşturduğu temalar. Yeni Medya Elektronik Dergisi, 7(2), 114-131. google scholar
21. Kirmani, A. R. (2022), Artificial Intelligence-Enabled Science Poetry. ACS Energy Letters, 8, 574-576. https://doi.org/10.1021/acsenergylett.2c02758 google scholar
22. Korkmaz, A., Aktürk, C., & Talan, T. (2023). Analyzing the user’s sentiments of ChatGPT using twitter data. Iraqi Journal for Computer Science and Mathematics, 4(2), 202-214. google scholar
23. Koubaa, A. (2023). GPT-4 vs. GPT-3.5: A concise showdown. Preprint. https://doi.org/10.20944/preprints202303.0422.v1 google scholar
24. Küçükkartal, H. K. (2020). Twitter’daki verilere metin madenciliği yöntemlerinin uygulanması. Eskişehir Türk Dünyası Uygulama ve Araştırma Merkezi Bilişim Dergisi, 1(2), 10-13. google scholar
25. Lampropoulos, G., Ferdig, R. E., & Kaplan-Rakowski, R. (2023). A social media data analysis of general and educational use of chatgpt: understanding emotional educators. Available at SSRN: https://ssrn.com/abstract=4468181 or http://dx.doi.org/10.2139/ssrn.4468181 google scholar
26. Li, L., Ma, Z., Fan, L., Lee, S., Yu, H., & Hemphill, L. (2023). ChatGPT in education: A discourse analysis of worries and concerns on social media, 1-35. arXiv preprint arXiv:2305.02201. google scholar
27. Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., ... & Ge, B. (2023). Summary of ChatGPT/GPT-4 research and perspective towards the future of large language models. 1-35. arXiv preprint arXiv: 2304.01852. https://doi.org/10.48550/arXiv.2304.01852. google scholar
28. Lund, B.D., & Wang, T. (2023). Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News, 40(3), 26-29. https://doi.org/10.1108/LHTN-01-2023-0009. google scholar
29. Metlek, S., & Kayaalp, K. (2020). Derin öğrenme ve destek vektör makineleri ile görüntüden cinsiyet tahmini. Düzce Üniversitesi Bilim ve Teknoloji Dergisi, 8(3), 2208-2228. google scholar
30. Munggaran, J. P., Alhafidz, A. A., Taqy, M., Agustini, D. A. R., & Munawir, M. (2023). Sentiment analysis of twitter users’ opinion data regarding the use of ChatGPT in education. Journal of Computer Engineering, Electronics and Information Technology, 2(2), 75-88. google scholar
31. Rachman F. H., Imamah, & Rintyarna B. S. (2022). Sentiment analysis of madura tourism in new normal era using text blob and KNN with hyperparameter tuning. 2021 International Seminar on Machine Learning, Optimization, and Data Science (ISMODE), 23-27. http://dx.doi.org/10.1109/ISMODE53584.2022.9742894 google scholar
32. Ulaş, M., & Karabay, B. (2020). Terör saldırılarını içeren büyük verinin makine öğrenmesi teknikleri ile analizi. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 32(1), 267-277. google scholar
33. Okoloegbo, C. A., Eze, U. F., Chukwudebe, G. A., & Nwokonkwo, O. C. (2022). Multilingual Cyberbullying Detector (CD) Application for Nigerian Pidgin and Igbo Language Corpus. In 2022 5th Information Technology for Education and Development (ITED), 1-6. https://doi.org/10.1109/ITED56637.2022.10051345 google scholar
34. OpenAI (2023). GPT-4 Technical Report. https://cdn.openai.com/papers/gpt-4.pdf. google scholar
35. Rahman, S., Jahan, N., Sadia, F., & Mahmud, I. (2023). Social crisis detection using twitter based text mining-a machine learning approach, BULLETIN OF ELECTRICAL ENGINEERING AND INFORMATICS, 12(2), 1069-1077. https://doi.org/10.11591/eei.v12i2.3957 google scholar
36. Santra, A. K., & Christy, C. J. (2012). Genetic algorithm and confusion matrix for document clustering. International Journal of Computer Science Issues (UCSI), 9(1), 322-328. google scholar
37. Shinde, P. P., & Shah, S. (2018). A review of machine learning and deep learning applications. In 2018 Fourth İnternational Conference on Computing Communication Control And Automation (ICCUBEA), 1-6. https://doi.org/10.1109/iccubea.2018.8697857 google scholar
38. Suh, B., Hong, L., Pirolli, P., & Chi, E. H. (2010). Want to be retweeted? Large scale analytics on factors impacting retweet in Twitter network. In 2010 Second İnternational Conference on Social Computing, 177-184. google scholar
39. Tekin, R., & Yaman, O. (2023). Akıllı ev sistemleri için XGBoost tabanlı saldırı tespit yöntemi. Journal of Intelligent Systems: Theory and Applications, 6(2), 152-158. https://doi.org/10.38016/jista.1075054 google scholar
40. Turan, A. K., & Polat, H. (2024). Yarı denetimli makine öğrenmesi yöntemini kullanarak müzik türlerinin tespiti. Gazi University Journal of Science Part C: Design and Technology, 12(1), 92-107. https://doi.org/10.29109/gujsc.1352477 google scholar
41. Veranyurt, Ü., Deveci, A., Esen, M. F., & Veranyurt, O. (2020). Makine öğrenmesi teknikleriyle hastalık sınıflandırması: Random forest, k-nearest neighbour ve adaboost algoritmaları uygulaması. Uluslararası Sağlık Yönetimi ve Stratejileri Araştırma Dergisi, 6(2), 275-286. google scholar
42. Verma, A., Mittal, V., & Dawn, S. (2019). FIND: Fake information and news detections using deep learning. In 2019 Twelfth İnternational Conference On Contemporary Computing (IC3), 1-7. https://doi.org/10.1109/IC3.2019.8844892 google scholar
43. Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A Survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731-5780. google scholar
44. Zahoor, S., & Rohilla, R. (2020). Twitter sentiment analysis using lexical or rule based approach: a case study. In 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), 537-542. google scholar
45. Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4), 1-25. google scholar