Exploring User Perceptions of ChatGPT: A Large-Scale Topic Modeling Analysis using LDA

Year 2025, Volume: 13 Issue: 3, 120 - 130, 30.09.2025

Murat Kılınç , Ahmet Ayaz

https://doi.org/10.21541/apjess.1677061

Abstract

Today, AI-based chatbots are widely used to improve the user experience and provide support in various areas. Especially advances in natural language processing (NLP) technologies have made these systems more intelligent and user-friendly. ChatGPT, developed by OpenAI, is one of the most popular applications in this field and serves millions of users. Users' experiences with ChatGPT are of great importance in determining the strengths and weaknesses of the model. In this study, Latent Dirichlet Allocation (LDA) method was used to analyze user feedback on ChatGPT. The main themes of user experiences were revealed by analyzing 250,000 user comments obtained through data scraping techniques on Google Play with topic modeling techniques. In our study, data cleaning and preprocessing steps were first applied, and then the comments were analyzed with LDA techniques. As a result of the analysis, user comments were grouped under certain themes and the most prominent themes were technical problems, feature requests, academic use, service quality and functionality. These themes help to understand ChatGPT's strengths and areas for improvement. For future studies, a more comprehensive evaluation by analyzing comments from different platforms is recommended.

Keywords

ChatGPT , Latent Dirichlet Allocation , Topic Modeling , User Feedback

References

• S. Kusal, S. Patil, J. Choudrie, K. Kotecha, S. Mishra, and A. Abraham, “AI-Based Conversational Agents: A Scoping Review from Technologies to Future Directions,” IEEE Access, vol. 10, pp. 92337–92356, 2022, doi: 10.1109/ACCESS.2022.3201144.
• L. Orynbay, G. Bekmanova, B. Yergesh, A. Omarbekova, A. Sairanbekova, and A. Sharipbay, “The role of cognitive computing in NLP,” Front. Comput. Sci., vol. 6, p. 1486581, Jan. 2025, doi: 10.3389/fcomp.2024.1486581.
• T. Adiguzel, H. Kaya, and F. Cansu, “Revolutionizing education with AI: Exploring the transformative potential of ChatGPT,” Contemp. Educ. Technol., vol. 15, p. ep429, Apr. 2023, doi: 10.30935/cedtech/13152.
• R. Bhayana, “Chatbots and Large Language Models in Radiology: A Practical Primer for Clinical and Research Applications,” Radiology, vol. 310, no. 1, p. e232756, Jan. 2024, doi: 10.1148/radiol.232756.
• G. Eysenbach, “The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation with ChatGPT and a Call for Papers,” JMIR Med. Educ., vol. 9, no. 1, p. e46885, Mar. 2023, doi: 10.2196/46885.
• A. Haleem, M. Javaid, and R. P. Singh, “An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges,” BenchCouncil Trans. Benchmarks Stand. Eval., vol. 2, no. 4, p. 100089, Oct. 2022, doi: 10.1016/j.tbench.2023.100089.
• L. Liu, H. An, P. Chen, and L. Ye, “A Contemporary Overview: Trends and Applications of Large Language Models on Mobile Devices,” 2024, doi: 10.48550/ARXIV.2412.03772.
• A. Kuzdeuov, O. Mukayev, S. Nurgaliyev, A. Kunbolsyn, and H. A. Varol, “ChatGPT for Visually Impaired and Blind,” 2024 Int. Conf. Artif. Intell. Inf. Commun. ICAIIC, pp. 722–727, Feb. 2024, doi: 10.1109/ICAIIC60209.2024.10463430.
• J. Dieber and S. Kirrane, “A novel model usability evaluation framework (MUsE) for explainable artificial intelligence,” Inf. Fusion, vol. 81, pp. 143–153, May 2022, doi: 10.1016/j.inffus.2021.11.017.
• B. Alsanousi, A. S. Albesher, H. Do, and S. Ludi, “Investigating the User Experience and Evaluating Usability Issues in AI-Enabled Learning Mobile Apps: An Analysis of User Reviews,” Int. J. Adv. Comput. Sci. Appl. IJACSA, vol. 14, no. 6, Art. no. 6, 30 2023, doi: 10.14569/IJACSA.2023.0140602.
• R. Albalawi, T. H. Yeap, and M. Benyoucef, “Using Topic Modeling Methods for Short-Text Data: A Comparative Analysis,” Front. Artif. Intell., vol. 3, Jul. 2020, doi: 10.3389/frai.2020.00042.
• O. Ozyurt, H. Özköse, and A. Ayaz, “Evaluating the latest trends of Industry 4.0 based on LDA topic model,” J. Supercomput., vol. 80, no. 13, pp. 19003–19030, Sep. 2024, doi: 10.1007/s11227-024-06247-x.
• M. Vanhala, C. Lu, J. Peltonen, S. Sundqvist, J. Nummenmaa, and K. Järvelin, “The usage of large data sets in online consumer behaviour: A bibliometric and computational text-mining–driven analysis of previous research,” J. Bus. Res., vol. 106, pp. 46–59, Jan. 2020, doi: 10.1016/j.jbusres.2019.09.009.
• S. Jin, G. Stokes, and C. Hamilton, “Empirical evidence of urban climate adaptation alignment with sustainable development: Application of LDA,” Cities, vol. 136, p. 104254, May 2023, doi: 10.1016/j.cities.2023.104254.
• G. Papadia, M. Pacella, and V. Giliberti, “Topic Modeling for Automatic Analysis of Natural Language: A Case Study in an Italian Customer Support Center,” Algorithms, vol. 15, no. 6, Art. no. 6, Jun. 2022, doi: 10.3390/a15060204.
• P. Pramanik and R. K. Jana, “Identifying research trends of machine learning in business: a topic modeling approach,” Meas. Bus. Excell., vol. 27, no. 4, pp. 602–633, Nov. 2023, doi: 10.1108/MBE-07-2021-0094.
• D. Blei, A. Ng, and M. Jordan, “Latent Dirichlet Allocation,” J. Mach. Learn. Res., no. 3, pp. 993–1022, 2003.
• D. Pagano and W. Maalej, “User feedback in the appstore: An empirical study,” in 2013 21st IEEE International Requirements Engineering Conference (RE), Jul. 2013, pp. 125–134. doi: 10.1109/RE.2013.6636712.
• E. Widnall et al., “User Perspectives of Mood-Monitoring Apps Available to Young People: Qualitative Content Analysis,” JMIR MHealth UHealth, vol. 8, no. 10, p. e18140, Oct. 2020, doi: 10.2196/18140.
• S. Okuboyejo and O. Koyejo, “Examining Users’ Concerns while Using Mobile Learning Apps,” Int. J. Interact. Mob. Technol. IJIM, vol. 15, no. 15, Art. no. 15, Aug. 2021, doi: 10.3991/ijim.v15i15.22345.
• J. Qiang, Z. Qian, Y. Li, Y. Yuan, and X. Wu, “Short Text Topic Modeling Techniques, Applications, and Performance: A Survey,” IEEE Trans. Knowl. Data Eng., vol. 34, no. 3, pp. 1427–1445, Mar. 2022, doi: 10.1109/TKDE.2020.2992485.
• Y. Zhai, X. Song, Y. Chen, and W. Lu, “A Study of Mobile Medical App User Satisfaction Incorporating Theme Analysis and Review Sentiment Tendencies,” Int. J. Environ. Res. Public. Health, vol. 19, no. 12, Art. no. 12, Jan. 2022, doi: 10.3390/ijerph19127466.
• Y. K. Oh and J.-M. Kim, “What Improves Customer Satisfaction in Mobile Banking Apps? An Application of Text Mining Analysis,” Asia Mark. J., vol. 23, no. 4, Feb. 2022, doi: 10.53728/2765-6500.1581.
• R. Egger and J. Yu, “A Topic Modeling Comparison Between LDA, NMF, Top2Vec, and BERTopic to Demystify Twitter Posts,” Front. Sociol., vol. 7, May 2022, doi: 10.3389/fsoc.2022.886498.
• A. B. Dieng, F. J. R. Ruiz, and D. M. Blei, “Topic Modeling in Embedding Spaces,” Trans. Assoc. Comput. Linguist., vol. 8, pp. 439–453, Jul. 2020, doi: 10.1162/tacl_a_00325.
• D. Antons and C. F. Breidbach, “Big Data, Big Insights? Advancing Service Innovation and Design with Machine Learning,” J. Serv. Res., vol. 21, no. 1, pp. 17–39, Feb. 2018, doi: 10.1177/1094670517738373.
• H. Cheng, S. Liu, W. Sun, and Q. Sun, “A Neural Topic Modeling Study Integrating SBERT and Data Augmentation,” Appl. Sci., vol. 13, no. 7, Art. no. 7, Jan. 2023, doi: 10.3390/app13074595.
• V. R. Hananto, U. Serdült, and V. Kryssanov, “A Text Segmentation Approach for Automated Annotation of Online Customer Reviews, Based on Topic Modeling,” Appl. Sci., vol. 12, no. 7, Art. no. 7, Jan. 2022, doi: 10.3390/app12073412.
• Y. An, H. Oh, and J. Lee, “Marketing Insights from Reviews Using Topic Modeling with BERTopic and Deep Clustering Network,” Appl. Sci., vol. 13, no. 16, Art. no. 16, Jan. 2023, doi: 10.3390/app13169443.
• S. Aydin, M. Karabacak, V. Vlachos, and K. Margetis, “Large language models in patient education: a scoping review of applications in medicine,” Front. Med., vol. 11, Oct. 2024, doi: 10.3389/fmed.2024.1477898.
• H. S. Jung, H. Lee, Y. S. Woo, S. Y. Baek, and J. H. Kim, “Expansive data, extensive model: Investigating discussion topics around LLM through unsupervised machine learning in academic papers and news,” PLOS ONE, vol. 19, no. 5, p. e0304680, May 2024, doi: 10.1371/journal.pone.0304680.
• T. L. Griffiths and M. Steyvers, “Finding scientific topics,” Proc. Natl. Acad. Sci., vol. 101, no. suppl_1, pp. 5228–5235, Apr. 2004, doi: 10.1073/pnas.0307752101.
• D. M. Blei, “Probabilistic topic models,” Commun ACM, vol. 55, no. 4, pp. 77–84, Apr. 2012, doi: 10.1145/2133806.2133826.
• I. Vayansky and S. A. P. Kumar, “A review of topic modeling methods,” Inf. Syst., vol. 94, p. 101582, Dec. 2020, doi: 10.1016/j.is.2020.101582.
• F. Gurcan and C. Kose, “Analysis of software engineering industry needs and trends: implications for education,” Int. J. Eng. Educ., vol. 33, no. 4, pp. 1361–1368, 2017.
• R. M. Amir Latif, M. Talha Abdullah, S. U. Aslam Shah, M. Farhan, F. Ijaz, and A. Karim, “Data Scraping from Google Play Store and Visualization of its Content for Analytics,” in 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), Jan. 2019, pp. 1–8. doi: 10.1109/ICOMET.2019.8673523.
• C. P. Paneru, C. Toșa, and A. K. M. Tarigan, “Exploring digital narratives: A comprehensive dataset of household energy app reviews,” Data Brief, vol. 51, p. 109835, Dec. 2023, doi: 10.1016/j.dib.2023.109835.
• H. Aldabbas, A. Bajahzar, M. Alruily, A. A. Qureshi, R. M. A. Latif, and M. Farhan, “Google Play Content Scraping and Knowledge Engineering using Natural Language Processing Techniques with the Analysis of User Reviews,” J. Intell. Syst., vol. 30, no. 1, pp. 192–208, Jan. 2021, doi: 10.1515/jisys-2019-0197.
• V. M. A. de Lima, A. F. de Araújo, and R. M. Marcacini, “Temporal dynamics of requirements engineering from mobile app reviews,” PeerJ Comput. Sci., vol. 8, p. e874, Mar. 2022, doi: 10.7717/peerj-cs.874.
• M. Al-mashhadani, K. Hussein, E. Khudir, and M. lyas, “Sentiment Analysis using Optimised Feature Sets in DifferentFacebook/Twitter Dataset Domains with Big Data,” Iraqi J. Comput. Sci. Math., vol. 3, no. 1, Jan. 2022, doi: 10.52866/ijcsm.2022.01.01.007.
• A. Iqbal, R. Amin, J. Iqbal, R. Alroobaea, A. Binmahfoudh, and M. Hussain, “Sentiment Analysis of Consumer Reviews Using Deep Learning,” Sustainability, vol. 14, no. 17, Art. no. 17, Jan. 2022, doi: 10.3390/su141710844.
• N. Mohamad Sham and A. Mohamed, “Climate Change Sentiment Analysis Using Lexicon, Machine Learning and Hybrid Approaches,” Sustainability, vol. 14, no. 8, Art. no. 8, Jan. 2022, doi: 10.3390/su14084723.
• G. Gonzalez et al., “Using Large-scale Social Media Analytics to Understand Patient Perspectives About Urinary Tract Infections: Thematic Analysis,” J. Med. Internet Res., vol. 24, no. 1, p. e26781, Jan. 2022, doi: 10.2196/26781.
• A. Thielmann, A. Reuter, Q. Seifert, E. Bergherr, and B. Säfken, “Topics in the Haystack: Enhancing Topic Quality through Corpus Expansion,” Comput. Linguist., vol. 50, no. 2, pp. 619–655, Jun. 2024, doi: 10.1162/coli_a_00506.
• K. Watanabe and A. Baturo, “Seeded Sequential LDA: A Semi-Supervised Algorithm for Topic-Specific Analysis of Sentences,” Soc. Sci. Comput. Rev., vol. 42, no. 1, pp. 224–248, Feb. 2024, doi: 10.1177/08944393231178605.
• P. Madzík and L. Falát, “State-of-the-art on analytic hierarchy process in the last 40 years: Literature review based on Latent Dirichlet Allocation topic modelling,” PLOS ONE, vol. 17, no. 5, p. e0268777, May 2022, doi: 10.1371/journal.pone.0268777.
• M. Steyvers and T. Griffiths, “Probabilistic Topic Models,” in Handbook of Latent Semantic Analysis, Psychology Press, 2007.
• R. Melisa et al., “Critical Thinking in the Age of AI: A Systematic Review of AI’s Effects on Higher Education,” Educ. Process Int. J., vol. 14, no. 1, 2025, doi: 10.22521/edupij.2025.14.31.
• J. E. Perez Perez, “The application of Gen-AI and creativity in the context of public education in frontier environments,” J. Enabling Technol., vol. 18, no. 4, pp. 223–231, Nov. 2024, doi: 10.1108/JET-05-2024-0030.
• A. Dominique Salamin, D. Russo, and D. Rueger, “ChatGPT, an Excellent Liar: How Conversational Agents’ Hallucinations Impact Learning and Teaching,” in Proceedings of the 7th International Conference on Teaching, Learning and Education, Acavent, Nov. 2023. doi: 10.33422/6th.iacetl.2023.11.100.
• S. S. Bhuyan et al., “Generative Artificial Intelligence Use in Healthcare: Opportunities for Clinical Excellence and Administrative Efficiency,” J. Med. Syst., vol. 49, no. 1, p. 10, Jan. 2025, doi: 10.1007/s10916-024-02136-1.
• L. Nissen, J. F. Rother, M. Heinemann, L. M. Reimer, S. Jonas, and T. Raupach, “A randomised cross-over trial assessing the impact of AI-generated individual feedback on written online assignments for medical students,” Med. Teach., pp. 1–7, Jan. 2025, doi: 10.1080/0142159X.2025.2451870.
• S. W. Bae, T. Chung, T. Zhang, A. K. Dey, and R. Islam, “Enhancing Interpretable, Transparent, and Unobtrusive Detection of Acute Marijuana Intoxication in Natural Environments: Harnessing Smart Devices and Explainable AI to Empower Just-In-Time Adaptive Interventions: Longitudinal Observational Study,” JMIR AI, vol. 4, p. e52270, Jan. 2025, doi: 10.2196/52270.
• E. Tokdemir Demi̇rel, “The Use and Perceptions Towards AI Tools for Academic Writing among University Students,” Innov. Lang. Teach. J., vol. 1, no. 1, pp. 1–20, 2021.
• F. D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Q., vol. 13, no. 3, p. 319, Sep. 1989, doi: 10.2307/249008.