Improving Hotel Review Rating Prediction with Transformer Models

Abstract

Online review platforms have become crucial decision-making tools in the hospitality industry, where automated sentiment analysis and rating prediction offer valuable insights for both businesses and consumers. This study investigates the performance of transformer-based language models for predicting hotel review ratings and examines the impact of oversampling techniques on model accuracy. We introduce a novel dataset of 68,785 English hotel reviews from TripAdvisor (2014-2023) in Turkey. Four transformer models, i.e., BERT, DistilBERT, RoBERTa, and DeBERTa, were systematically compared using multiple perspectives. Results show DeBERTa achieves the highest performance among all evaluated models. Random oversampling (ROS) significantly improved classification performance, with F1-scores increasing from 62% to 81% and accuracy from 76% to over 82% across all models. The oversampling approach effectively addressed class imbalance while preserving semantic information, enabling better distinction between rating categories. Through quantitative and qualitative analysis, including the embedding of visualization and SHAP-based interpretability studies, we demonstrate that transformer models effectively capture sentiment patterns. However, they remain sensitive to mixed sentiments and linguistic subtleties. This work contributes a novel dataset, a systematic comparison of four transformer models, and empirical evidence of oversampling effectiveness in sentiment analysis.

Keywords

• Sentiment analysis
• Transformer models
• Rating prediction
• Oversampling

References

1. O. Ciftci, K. Berezina, M. Cavusoglu, and C. Cobanoglu, “Winning the battle: The importance of price and online reviews for hotel selection,” Adv. Hospitality Tourism Res., vol. 8, no. 1, pp. 177–202, Jun. 2020, doi: 10.30519/ahtr.528150.
2. M. Suwal, P. Neupane, and G. D. Pant, “Online review on hotel booking decision: Consumer view,” Int. J. Atharva, vol. 3, no. 1, pp. 133–150, Mar. 2025, doi: 10.3126/ija.v3i1.76724.
3. P. S. Ghatora, S. E. Hosseini, S. Pervez, M. J. Iqbal, and N. Shaukat, “Sentiment analysis of product reviews using machine learning and pre-trained LLM,” Big Data Cogn. Comput., vol. 8, no. 12, Art. no. 199, Dec. 2024, doi: 10.3390/bdcc8120199.
4. N. Malik and M. Bilal, “Natural language processing for analyzing online customer reviews: A survey, taxonomy, and open research challenges,” PeerJ Comput. Sci., vol. 10, Art. no. e2203, Aug. 2024, doi: 10.20944/preprints202312.2210.v1.
5. J. Hartmann, M. Heitmann, C. Siebert, and C. Schamp, “More than a feeling: Accuracy and application of sentiment analysis,” Int. J. Res. Marketing, vol. 40, no. 1, pp. 75–87, Mar. 2023, doi: 10.1016/j.ijresmar.2022.05.005.
6. R. Obiedat et al., “Sentiment analysis of customers’ reviews using a hybrid evolutionary SVM-based approach in an imbalanced data distribution,” IEEE Access, vol. 10, pp. 22260–22273, Mar. 2022, doi: 10.1109/ACCESS.2022.3149482.
7. W. Zhou, Y. Wang, Y. Qu, and L. Li, “Automating app review classification based on extended semantic,” in Proc. 9th Int. Conf. Dependable Syst. Appl. (DSA), Aug. 2022, pp. 106–115, doi: 10.1109/DSA56465.2022.00022.
8. Y. C. A. P. Reddy, S. P. P. Sagar, R. P. Kalyan, and N. S. Charan, “Classification of hotel reviews using machine learning techniques,” in Proc. 8th Int. Conf. Smart Struct. Syst. (ICSSS), Apr. 2022, pp. 1–5, doi: 10.1109/ICSSS54381.2022.9782215.

9. A. R. Simarmata and M. Zakariyah, “Sentiment analysis of hotel reviews using support vector machine,” Indonesian J. Comput. Sci., vol. 12, no. 5, pp. 2603–2614, Nov. 2023, doi: 10.33022/ijcs.v12i5.3405.
10. S. Yordanova and D. Kabakchieva, “Sentiment classification of hotel reviews in social media with decision tree learning,” Int. J. Comput. Appl., vol. 158, no. 5, pp. 1–7, Jan. 2017, doi: 10.5120/ijca2017912806.
11. S. Pratap, A. R. Aranha, D. Kumar, G. Malhotra, A. P. N. Iyer, and S. S. S., “The fine art of fine-tuning: A structured review of advanced LLM fine-tuning techniques,” Natural Lang. Process. J., vol. 11, Art. no. 100144, Jun. 2025, doi: 10.1016/j.nlp.2025.100144.
12. Y. Gui, X. Yan, P. Yin, H. Yang, and J. Cheng, “SPT: Fine-tuning transformer-based language models efficiently with sparsification,” arXiv preprint arXiv:2312.10365, Dec. 2023.
13. H. Wang, J. Li, H. Wu, E. Hovy, and Y. Sun, “Pre-trained language models and their applications,” Engineering, vol. 25, pp. 51–65, Jun. 2023, doi: 10.1016/j.eng.2022.04.024.
14. Y. G. Pramudya and A. Alamsyah, “Hotel reviews classification and review-based recommendation model construction using BERT and RoBERTa,” in Proc. 6th Int. Conf. Inf. Commun. Technol. (ICOIACT), Nov. 2023, pp. 437–442, doi: 10.1109/ICOIACT59844.2023.10455890.
15. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol. (NAACL-HLT), Jun. 2019, pp. 4171–4186.
16. Y. Liu et al., “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, Jul. 2019.
17. Y. Yuan, “DistilBERT hotel rating prediction model based on an ensemble learning framework,” in Proc. 3rd Int. Conf. Electron. Inf. Technol. (EIT), Sep. 2024, pp. 763–769, doi: 10.1109/EIT63098.2024.10762068.
18. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, Oct. 2019.
19. M. S. Asyaky, M. Al-Husaini, and H. H. Lukmana, “Sentiment analysis on short social media texts using DistilBERT,” J. Comput. Netw. Archit. High Perform. Comput., vol. 7, no. 2, pp. 524–533, May 2025, doi: 10.47709/cnahpc.v7i2.5836.
20. M. Chen, H. Xu, Y. Wu, and J. Wu, “Sentiment analysis of hotel reviews based on BERT and XGBoost,” in Proc. 3rd Int. Conf. Comput. Technol. (ICCTech), Feb. 2024, pp. 11–15, doi: 10.1109/ICCTech61708.2024.00011.
21. V. Dogra, S. Verma, A. Singh, Kavita, M. N. Talib, and M. Humayun, “Banking news-events representation and classification with a novel hybrid model using DistilBERT and rule-based features,” Turkish J. Comput. Math. Educ., vol. 12, no. 10, pp. 3039–3054, Apr. 2021.
22. A. P. Ratnasari and R. Nur’aini, “Performance of random oversampling, random undersampling, and SMOTE-NC methods in handling imbalanced class in classification models,” Int. J. Sci. Res. Manag., vol. 12, no. 04, pp. 494–501, Apr. 2024, doi: 10.18535/ijsrm/v12i04.m03.
23. R. Mohammed, J. Rawashdeh, and M. Abdullah, “Machine learning with oversampling and undersampling techniques: Overview study and experimental results,” in Proc. 11th Int. Conf. Inf. Commun. Syst. (ICICS), Apr. 2020, pp. 243–248, doi: 10.1109/ICICS49469.2020.239556.
24. M. M. Ahsan, M. S. Ali, and Z. Siddique, “Enhancing and improving the performance of imbalanced class data using novel GBO and SSG: A comparative analysis,” Neural Netw., vol. 173, Art. no. 106157, May 2024, doi: 10.1016/j.neunet.2024.106157.
25. S. García and F. Herrera, “Evolutionary undersampling for classification with imbalanced datasets: Proposals and taxonomy,” Evol. Comput., vol. 17, no. 3, pp. 275–306, Sep. 2009, doi: 10.1162/evco.2009.17.3.275.
26. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953.
27. M. Mujahid et al., “Data oversampling and imbalanced datasets: An investigation of performance for machine learning and feature engineering,” J. Big Data, vol. 11, Art. no. 87, Jun. 2024, doi: 10.1186/s40537-024-00943-4.
28. M. Goyal and Q. H. Mahmoud, “A systematic review of synthetic data generation techniques using generative AI,” Electronics, vol. 13, no. 17, Art. no. 3509, Sep. 2024, doi: 10.3390/electronics13173509.
29. A. P. Ratnasari and R. Nur’aini, “Performance of random oversampling, random undersampling, and SMOTE-NC methods in handling imbalanced class in classification models,” Int. J. Sci. Res. Manag., vol. 12, no. 04, pp. 494–501, Apr. 2024, doi: 10.18535/ijsrm/v12i04.m03.
30. Z. Zhang, Z. Li, J. Zhu, Z. Guo, B. Shi, and B. Hu, “Enhancing user sequence representation with cross-view collaborative learning for depression detection on Sina Weibo,” Knowl.-Based Syst., vol. 293, Art. no. 111650, Jun. 2024, doi: 10.1016/j.knosys.2024.111650.
31. A. A. Khan, O. Chaudhari, and R. Chandra, “A review of ensemble learning and data augmentation models for class imbalanced problems: Combination, implementation and evaluation,” Expert Syst. Appl., vol. 244, Art. no. 122778, Jun. 2024, doi: 10.1016/j.eswa.2023.122778.
32. D. A. Sani, “A random oversampling and BERT-based model approach for handling imbalanced data in essay answer correction,” J. Infotel, vol. 16, no. 4, pp. 729–739, Dec. 2024, doi: 10.20895/infotel.v16i4.1224.
33. H. Rathpisey and T. B. Adji, “Handling imbalance issue in hate speech classification using sampling-based methods,” in Proc. 5th Int. Conf. Sci. Inf. Technol. (ICSITech), Oct. 2019, pp. 193–198, doi: 10.1109/ICSITech46713.2019.8987500.
34. C. W. Schmidt et al., “Tokenization is more than compression,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), Nov. 2024, pp. 678–702.
35. X. Song, A. Salcianu, Y. Song, D. Dopson, and D. Zhou, “Fast WordPiece tokenization,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), Nov. 2021, pp. 2089–2103.
36. L. Kozma and J. Voderholzer, “Theoretical analysis of Byte-Pair Encoding,” arXiv preprint arXiv:2411.08671, Nov. 2024.
37. W. Zhang, W. Wei, W. Wang, L. Jin, and Z. Cao, “Reducing BERT computation by padding removal and curriculum learning,” in Proc. IEEE Int. Symp. Perform. Anal. Syst. Softw. (ISPASS), Mar. 2021, pp. 90–92, doi: 10.1109/ISPASS51385.2021.00025.