Document Classification with Contextually Enriched Word Embeddings

Abstract

The text classification task has a wide range of application domains for distinct purposes, such as the classification of articles, social media posts, and sentiments. As a natural language processing application, machine learning and deep learning techniques are intensively utilized in solving such challenges. One common approach is employing the discriminative word features comprising Bag-of-Words and n-grams to conduct text classification experiments. The other powerful approach is exploiting neural network-based (specifically deep learning models) through either sentence, word, or character levels. In this study, we proposed a novel approach to classify documents with contextually enriched word embeddings powered by the neighbor words accessible through the trigram word series. In the experiments, a well-known web of science dataset is exploited to demonstrate the novelty of the models. Consequently, we built various models constructed with and without the proposed approach to monitor the models' performances. The experimental models showed that the proposed neighborhood-based word embedding enrichment has decent potential to use in further studies.

Keywords

• Text classification
• Deep Learning
• LSTM
• Word2Vec
• N-grams

Supporting Institution

The authors received no financial support for the research, authorship, and/or publication of this article.

Ethical Statement

The authors have no conflicts of interest to disclose.

References

1. [1] A. J. Trappey, F.-C. Hsu, C. V. Trappey, and C.-I. Lin, “Development of a patent document classification and search platform using a backpropagation network,” Expert Systems with Applications, vol. 31, no. 4, pp. 755–765, 2006.
2. [2] G. Aghila et al., “A survey of nan” ive bayes machine learning approach in text document classification,” arXiv preprint arXiv:1003.1795, 2010.
3. [3] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv preprint arXiv:1607.01759, 2016.
4. [4] Q. Chen and M. Sokolova, “Specialists, scientists, and sentiments: Word2vec and doc2vec in analysis of scientific and medical texts,” SN Computer Science, vol. 2, pp. 1–11, 2021.
5. [5] G. Bakal and O. Abar, “On comparative classification of relevant covid- 19 tweets,” in 2021 6th International Conference on Computer Science and Engineering (UBMK). IEEE, 2021, pp. 287–291.
6. [6] B. J. Marafino, J. M. Davies, N. S. Bardach, M. L. Dean, and R. A. Dudley, “N-gram support vector machines for scalable procedure and diagnosis classification, with applications to clinical free text data from the intensive care unit,” Journal of the American Medical Informatics Association, vol. 21, no. 5, pp. 871–875, 2014.
7. [7] A. Graves, S. Fern´andez, and J. Schmidhuber, “Bidirectional lstm networks for improved phoneme classification and recognition,” in Artificial Neural Networks: Formal Models and Their Applications–ICANN 2005: 15th International Conference, Warsaw, Poland, September 11-15, 2005. Proceedings, Part II 15. Springer, 2005, pp. 799–804.
8. [8] A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” Neural networks, vol. 18, no. 5-6, pp. 602–610, 2005.

9. [9] L. Xiao, G. Wang, and Y. Zuo, “Research on patent text classification based on word2vec and lstm,” in 2018 11th International Symposium on Computational Intelligence and Design (ISCID), vol. 1. IEEE, 2018, pp. 71–74.
10. [10] C. C. Aggarwal and C. C. Aggarwal, “Machine learning for text: An introduction,” Machine learning for text, pp. 1–16, 2018.
11. [11] P. Milhorat, S. Schl¨ogl, G. Chollet, J. Boudy, A. Esposito, and G. Pelosi, “Building the next generation of personal digital assistants,” in 2014 1st international conference on advanced technologies for signal and image processing (atsip). IEEE, 2014, pp. 458–463.
12. [12] H.-H. Chiang, “A comparison between teacher-led and online text-tospeech dictation for students’ vocabulary performance.” English Language Teaching, vol. 12, no. 3, pp. 77–93, 2019.
13. [13] E. Adamopoulou and L. Moussiades, “Chatbots: History, technology, and applications,” Machine Learning with Applications, vol. 2, p. 100006, 2020.
14. [14] D. Martinez, O. Plchot, L. Burget, O. Glembek, and P. Matˇejka, “Language recognition in ivectors space,” in Twelfth annual conference of the international speech communication association, 2011.
15. [15] T. Pedersen and R. Bruce, “Distinguishing word senses in untagged text,” arXiv preprint cmp-lg/9706008, 1997.
16. [16] M. Bevilacqua, T. Pasini, A. Raganato, and R. Navigli, “Recent trends in word sense disambiguation: A survey,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. International Joint Conference on Artificial Intelligence, Inc, 2021.
17. [17] D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes, vol. 30, no. 1, pp. 3–26, 2007.
18. [18] W. Medhat, A. Hassan, and H. Korashy, “Sentiment analysis algorithms and applications: A survey,” Ain Shams engineering journal, vol. 5, no. 4, pp. 1093–1113, 2014.
19. [19] E. Reiter and R. Dale, “Building applied natural language generation systems,” Natural Language Engineering, vol. 3, no. 1, pp. 57–87, 1997.
20. [20] B. Kolukisa, B. K. Dedeturk, B. A. Dedeturk, A. Gulsen, and G. Bakal, “A comparative analysis on medical article classification using text mining & machine learning algorithms,” in 2021 6th International Conference on Computer Science and Engineering (UBMK). IEEE, 2021, pp. 360–365.
21. [21] A. Adhikari, A. Ram, R. Tang, and J. Lin, “Rethinking complex neural network architectures for document classification,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4046–4051.