Sentiment Analysis on GPT-4 with Comparative Models Using Twitter Data

Year 2024, Volume: 8 Issue: 1, 23 - 33, 28.06.2024

Mustafa Özel , Özlem Çetinkaya Bozkurt

https://doi.org/10.26650/acin.1418834

Abstract

Every day, people from all over the world use Twitter to talk about many different topics using hashtags. Since ChatGPT was launched, researchers have been studying how people perceive it in society. This research aims to find out what Turkish Twitter users think about OpenAI’s latest AI model called Generative Pre-trained Transformer 4 (GPT-4). The quantitative data used in this study consist of hashtags on the topic of GPT-4 and involve 2,978 tweets on this topic that were shared on Twitter between March 14-April 9, 2023. The study uses TextBlob sentiment scores to classify the tweets and support vector machines, logistic regression, XGBoost, and random forest algorithms to classify the sentiment of the dataset. The results from the logistic regression, XGBoost, and support vector methods are in close alignment. All parameter findings indicate dependable machine learning, emphasizing the models’ success in classifying tweet sentiment.

Keywords

Sentiment analysis, social media, Twitter, natural language processing

References

• Abdullah M., Madain A., & Jararweh Y. (2022). ChatGPT: fundamentals, applications and social impacts. 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS). 1-8. http://dx.doi.org/10.1109/SNAMS58071.2022.10062688 google scholar
• Ağralı, Ö., & Aydın, Ö. (2021). Tweet classification and sentiment analysis on metaverse related messages,. Journal of Metaverse, 1(1), 25-30. google scholar
• Aydın, C. (2018). makine öğrenmesi algoritmalari kullanilarak itfaiye istasyonu ihtiyacinin siniflandirilmasi. Avrupa Bilim ve Teknoloji Dergisi, (14), 169-175. google scholar
• Aydın, Ö., & Karaarslan, E. (2023). Is ChatGPT leading generative AI? What is beyond expectations?. 1-23. Social Science Research Network (SSRN): https://ssrn.com/abstract=4341500 or http://dx.doi.org/10.2139/ssrn.4341500. google scholar
• Ayhan, S., & Erdoğmuş, Ş. (2014). Destek vektör makineleriyle siniflandirma problemlerinin çözümü için çekirdek fonksiyonu seçimi. Eskişehir Osmangazi Üniversitesi İktisadi ve İdari Bilim Dergisi, 9(1), 175-198. google scholar
• Agarwal, B., Mittal, N., Bansal, P., & Garg, S. (2015). Sentiment analysis using common-sense and context information. Computational İntelligence and Neuroscience, 2015, 1-9. http://dx.doi.org/10.1155/2015/715730 google scholar
• Bayram, İ., & Turan, A. (2022). Türkiye’de kripto para farkindaliği ve tutumu: duygu analizi ve istatistiksel analiz ile bir değerlendirme. Yönetim Bilişim Sistemleri Dergisi, 8(2), 20-35. https://dergipark.org.tr/en/pub/ybs/issue/75943/1197985 google scholar
• Bengesi S., Oladunni T., Olusegun R., & Audu H. (2023). A Machine Learning-Sentiment Analysis on Monkeypox Outbreak: An Extensive Dataset to Show the Polarity of Public Opinion From Twitter Tweets, IEEE Access, 11( 11811-11826). http://dx.doi.org/10.1109/ACCESS.2023.3242290 google scholar
• Bircan, H. (2004). Lojistik regresyon analizi: Tıp verileri üzerine bir uygulama. Kocaeli Üniversitesi Sosyal Bilimler Dergisi, (8), 185-208. google scholar
• Biltawi, M., Etaiwi, W., Tedmori, S., Hudaib, A., & Awajan, A. (2016). Sentiment classification techniques for arabic language: A survey. 7th International Conference on Information and Communication Systems (ICICS), 339-346. google scholar
• Botchu, R., & Iyengar, K. P. (2023). Will ChatGPT drive radiology in the future?. Indian Journal of Radiology and Imaging. https://doi.org/10.1055/s-0043-1769591 google scholar
• Bruns, A., & Burgess, J. E. (2011). The use of Twitter hashtags in the formation of ad hoc publics. Paper presented at the 6th European Consortium for Political Research General Conference, 25 - 27 August 2011, University of Iceland, Reykjavik. http://eprints.qut.edu.au/46515/ google scholar
• Dandekar, A. R., Shrotri, A.P., Hargude, N.V., Awate, P.P., & Patil, N.D. (2018,August). The rise of social media and its impact. Internatıonal Journal of Research And Analytıcal Revıews 5(3), 882-886. google scholar
• Diyasa, I. G. S. M., Mandenni, N. M. I. M., Fachrurrozi, M. I., Pradika, S. I., Manab, K. R. N., & Sasmita, N. R. (2021). Twitter sentiment analysis as an evaluation and service base on python textblob. In IOP Conference Series: Materials Science and Engineering, 1125(1), 1-12. https://doi.org/10.1088/1757-899X/1125/1/012034 google scholar
• Doğan, G. (2022). Makine öğrenmesi algoritmalari ile betonarme kirişlerin burulma momenti tahmini. El-Cezeri El-Cezerî Fen ve Mühendislik Dergisi, 9(2), 912-924. google scholar
• Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023). Investigating code generation performance of chatgpt with crowdsourcing social data. In Proceedings of the 47th IEEE Computer Software and Applications Conference, 1-10. google scholar
• Hardeniya, T., & Borikar, D. A. (2016). Dictionary based approach to sentiment analysis-a review. International Journal of Advanced Engineering, Management and Science, 2(5), 317-322. google scholar
• Kahya, A. N. (2021). Wikipedia’daki verilere metin madenciliği yöntemlerinin uygulanmasi. Eskişehir Türk Dünyası Uygulama ve Araştırma Merkezi (ESTUDAM) Bilişim Dergisi, 2(1), 11-14. google scholar
• Kaur C., & Sharma A. (2022). Social issues sentiment analysis using python. 2020 5th International Conference on Computing, Communication and Security (ICCCS), 1-6. http://dx.doi.org/10.1109/ICCCS49678.2020.9277251 google scholar
• Keskin, E. K. (2023). Yapay zekâ sohbet robotu ChatGPT ve Türkiye internet gündeminde oluşturduğu temalar. Yeni Medya Elektronik Dergisi, 7(2), 114-131. google scholar
• Kirmani, A. R. (2022), Artificial Intelligence-Enabled Science Poetry. ACS Energy Letters, 8, 574-576. https://doi.org/10.1021/acsenergylett.2c02758 google scholar
• Korkmaz, A., Aktürk, C., & Talan, T. (2023). Analyzing the user’s sentiments of ChatGPT using twitter data. Iraqi Journal for Computer Science and Mathematics, 4(2), 202-214. google scholar
• Koubaa, A. (2023). GPT-4 vs. GPT-3.5: A concise showdown. Preprint. https://doi.org/10.20944/preprints202303.0422.v1 google scholar
• Küçükkartal, H. K. (2020). Twitter’daki verilere metin madenciliği yöntemlerinin uygulanması. Eskişehir Türk Dünyası Uygulama ve Araştırma Merkezi Bilişim Dergisi, 1(2), 10-13. google scholar
• Lampropoulos, G., Ferdig, R. E., & Kaplan-Rakowski, R. (2023). A social media data analysis of general and educational use of chatgpt: understanding emotional educators. Available at SSRN: https://ssrn.com/abstract=4468181 or http://dx.doi.org/10.2139/ssrn.4468181 google scholar
• Li, L., Ma, Z., Fan, L., Lee, S., Yu, H., & Hemphill, L. (2023). ChatGPT in education: A discourse analysis of worries and concerns on social media, 1-35. arXiv preprint arXiv:2305.02201. google scholar
• Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., ... & Ge, B. (2023). Summary of ChatGPT/GPT-4 research and perspective towards the future of large language models. 1-35. arXiv preprint arXiv: 2304.01852. https://doi.org/10.48550/arXiv.2304.01852. google scholar
• Lund, B.D., & Wang, T. (2023). Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News, 40(3), 26-29. https://doi.org/10.1108/LHTN-01-2023-0009. google scholar
• Metlek, S., & Kayaalp, K. (2020). Derin öğrenme ve destek vektör makineleri ile görüntüden cinsiyet tahmini. Düzce Üniversitesi Bilim ve Teknoloji Dergisi, 8(3), 2208-2228. google scholar
• Munggaran, J. P., Alhafidz, A. A., Taqy, M., Agustini, D. A. R., & Munawir, M. (2023). Sentiment analysis of twitter users’ opinion data regarding the use of ChatGPT in education. Journal of Computer Engineering, Electronics and Information Technology, 2(2), 75-88. google scholar
• Rachman F. H., Imamah, & Rintyarna B. S. (2022). Sentiment analysis of madura tourism in new normal era using text blob and KNN with hyperparameter tuning. 2021 International Seminar on Machine Learning, Optimization, and Data Science (ISMODE), 23-27. http://dx.doi.org/10.1109/ISMODE53584.2022.9742894 google scholar
• Ulaş, M., & Karabay, B. (2020). Terör saldırılarını içeren büyük verinin makine öğrenmesi teknikleri ile analizi. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 32(1), 267-277. google scholar
• Okoloegbo, C. A., Eze, U. F., Chukwudebe, G. A., & Nwokonkwo, O. C. (2022). Multilingual Cyberbullying Detector (CD) Application for Nigerian Pidgin and Igbo Language Corpus. In 2022 5th Information Technology for Education and Development (ITED), 1-6. https://doi.org/10.1109/ITED56637.2022.10051345 google scholar
• OpenAI (2023). GPT-4 Technical Report. https://cdn.openai.com/papers/gpt-4.pdf. google scholar
• Rahman, S., Jahan, N., Sadia, F., & Mahmud, I. (2023). Social crisis detection using twitter based text mining-a machine learning approach, BULLETIN OF ELECTRICAL ENGINEERING AND INFORMATICS, 12(2), 1069-1077. https://doi.org/10.11591/eei.v12i2.3957 google scholar
• Santra, A. K., & Christy, C. J. (2012). Genetic algorithm and confusion matrix for document clustering. International Journal of Computer Science Issues (UCSI), 9(1), 322-328. google scholar
• Shinde, P. P., & Shah, S. (2018). A review of machine learning and deep learning applications. In 2018 Fourth İnternational Conference on Computing Communication Control And Automation (ICCUBEA), 1-6. https://doi.org/10.1109/iccubea.2018.8697857 google scholar
• Suh, B., Hong, L., Pirolli, P., & Chi, E. H. (2010). Want to be retweeted? Large scale analytics on factors impacting retweet in Twitter network. In 2010 Second İnternational Conference on Social Computing, 177-184. google scholar
• Tekin, R., & Yaman, O. (2023). Akıllı ev sistemleri için XGBoost tabanlı saldırı tespit yöntemi. Journal of Intelligent Systems: Theory and Applications, 6(2), 152-158. https://doi.org/10.38016/jista.1075054 google scholar
• Turan, A. K., & Polat, H. (2024). Yarı denetimli makine öğrenmesi yöntemini kullanarak müzik türlerinin tespiti. Gazi University Journal of Science Part C: Design and Technology, 12(1), 92-107. https://doi.org/10.29109/gujsc.1352477 google scholar
• Veranyurt, Ü., Deveci, A., Esen, M. F., & Veranyurt, O. (2020). Makine öğrenmesi teknikleriyle hastalık sınıflandırması: Random forest, k-nearest neighbour ve adaboost algoritmaları uygulaması. Uluslararası Sağlık Yönetimi ve Stratejileri Araştırma Dergisi, 6(2), 275-286. google scholar
• Verma, A., Mittal, V., & Dawn, S. (2019). FIND: Fake information and news detections using deep learning. In 2019 Twelfth İnternational Conference On Contemporary Computing (IC3), 1-7. https://doi.org/10.1109/IC3.2019.8844892 google scholar
• Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A Survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731-5780. google scholar
• Zahoor, S., & Rohilla, R. (2020). Twitter sentiment analysis using lexical or rule based approach: a case study. In 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), 537-542. google scholar
• Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4), 1-25. google scholar