Performance Evaluation of a Pretrained BERT Model for Automatic Text Classification

Abstract

This study presents a pre-trained BERT model application on texts that are extracted from website URLs automatically to classify texts according to the industry. With the aim of doing so, the related dataset is first obtained from different kinds of websites by web scraping. Then, the dataset is cleaned and labeled with the relevant industries among 42 different categories. The pre-trained BERT model which was trained on 101.000 advertisement texts in one of our previous ad text classification studies is used to classify texts. Classification performance metrics are then used to evaluate the pre-trained BERT model on the test set and 0.98 average accuracy and 0.67 average F1 score for different 12 categories are obtained. The method can be used to test the compatibility of texts to be used in online advertising networks with the advertiser's industry. In this way, the suitability of the texts, which is an important component in determining the quality of online advertising, within the industry will be tested automatically.

Keywords

• digital marketing
• ad text
• natural language processing
• text classification
• bidirectional encoder representations from transformers

References

1. [1] Desai, Vaibhava, and B. Vidyapeeth. "Digital marketing: A review." International Journal of Trend in Scientific Research and Development 5.5 (2019): 196-200.
2. [2] Z, A. and Adali, E., “Opinion mining and sentiment analysis for contex-tual online-advertisement,” in 2016 IEEE 10th International Conference on Application of Information and Communication Technologies (AICT). IEEE, 2016, pp. 1–3.
3. [3] “Reklam kalitesi hakkında - Google Ads Yardım,” https://support.google. com/google-ads/answer/156066?hl=tr&ref topic=10549746, May 2021.
4. [4] Ş. Ozan and D. E. Taşar, "Auto-tagging of Short Conversational Sentences using Natural Language Processing Methods," 2021 29th Signal Processing and Communications Applications Conference (SIU), 2021, pp. 1-4, doi: 10.1109/SIU53274.2021.9477994
5. [5] Rønningstad, E., “Targeted sentiment analysis for norwegian text,” 2020.
6. [6] Özdil, U., Arslan, B., Taşar, D. E., Polat, G., & Ozan, Ş. (2021, September). Ad Text Classification with Bidirectional Encoder Representations. In 2021 6th International Conference on Computer Science and Engineering (UBMK) (pp. 169-173). IEEE.
7. [7] Deng, L., & Yu, D. (2014). Deep Learning: Methods and Applications. Foundations and Trends in Signal Processing (Cilt 7, s. 197-387).
8. [8] Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G. Z. (2017). “Deep learning for health informatics”. IEEE journal of Biomedical and Health Informatics 21(1), 4-21.

9. [9] Mikolov, C. (2013). Mikolov T., Chen K., Corrado G., Dean J. Efficient estimation of word representations in vector space, CoRR.
10. [10] Pennington, J., Socher, R., & Manning, C. D. (2014, October). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).
11. [11] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
12. [12] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
13. [13] Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J. (2021). Deep learning--based text classification: a comprehensive review. ACM computing surveys (CSUR), 54(3), 1-40.
14. [14] Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
15. [15] Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
16. [16] Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
17. [17] Liu, Y., Ott, M., & Goyal, N. (2019). Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. Computing Research Repository.
18. [18] Lee, J. Y., & Dernoncourt, F. (2016). “Sequential short-text classification with recurrent and convolutional neural networks”. arXiv preprint arXiv:1603.03827.
19. [19] Kim, Y. (2014). “Convolutional neural networks for sentence classification”. arXiv preprint arXiv:1408.5882.
20. [20] Olsson, F., Sahlgren, M., Abdesslem, F. B., Ekgren, A., & Eck, K. (2020, May). Text categorization for conflict event annotation. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020 (pp. 19-25).
21. [21] Gonzalez-Carvajal, S. and Garrido-Merchan, E. C., “Comparing bert against traditional machine learning text classification,” arXiv preprint arXiv:2005.13012, 2020
22. [22] Reimers, N., & Gurevych, I. (2020). Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813.
23. [23] Cruz, J. C. B., & Cheng, C. (2020). Establishing baselines for text classification in low-resource languages. arXiv preprint arXiv:2005.02068.
24. [24] Şükrü, O. Z. A. N., et al. "BERT Modeli'nin Sınıflandırma Doğruluğunun Sıfır-Atış Öğrenmesi ile Artırılması." Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi 14.2 (2021): 99-108.
25. [25] Özdil, U., Arslan, B., Taşar, D. E., Polat, G., & Ozan, Ş. (2021). Ad Text Classification with transformer-based natural language processing methods. arXiv preprint arXiv:2106.10899.
26. [26] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K., “Bert: Pre-trainingof deep bidirectional transformers for language understanding,” arXivpreprint arXiv:1810.04805, 2018.
27. [27] Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., & Zhong, J. (2021, June). Attention is all you need in speech separation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 21-25). IEEE.
28. [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez,A. N., Kaiser, L., and Polosukhin, I., “Attention is all you need,” arXivpreprint arXiv:1706.03762, 2017.
29. [29] Loodos., “loodos/bert-base-turkish-uncased · hugging face,” https://github.com/Loodos/turkish-languagemodels, Aug. 2020.