A LITERATURE STUDY ON DEEP LEARNING APPLICATIONS IN NATURAL LANGUAGE PROCESSING

Abstract

Deep learning is an important and recent topic of artificial intelligence and machine learning areas. Especially in recent years, the number of studies proposing different deep learning methods and applying these methods on different problems is increasing. These methods have also been used at various subareas of natural language processing extensively, and are still being used. In this survey paper, firstly, classification of deep learning techniques is presented and then important studies about deep learning approaches for natural language processing problems are discussed. It is expected that the number and prevalence of both theoretical studies and studies with practical applications on deep learning and on deep learning solutions to natural language processing problems are going to increase. Therefore it is considered that our study will be an important Turkish resource on the topic of deep learning applications for natural language processing.

Keywords

• Natural Language Processing,Text Mining,Deep Learning,Machine Learning

DOĞAL DİL İŞLEMEDE DERİN ÖĞRENME UYGULAMALARI ÜZERİNE BİR LİTERATÜR ÇALIŞMASI

Abstract

Derin öğrenme, yapay zekâ ve makine öğrenmesi alanlarının önemli ve güncel bir konusu haline gelmiştir. Özellikle son yıllarda, farklı derin öğrenme yöntemleri öneren çalışmaların ve mevcut yöntemleri değişik problemler üzerinde uygulayan çalışmaların sayıları hızla artmaktadır. Doğal dil işlemenin çeşitli alt alanlarında da bu yöntemler yaygın olarak kullanılmış ve halen kullanılmaktadır. Bu derleme çalışmasında, ilk olarak derin öğrenme yöntemlerinin bir sınıflandırması sunulmuş, ardından da doğal dil işleme problemlerine derin öğrenme yaklaşımlarının sunulduğu önemli çalışmalar incelenmiştir. Derin öğrenme ve doğal dil işleme problemlerinin çözümü amacıyla derin öğrenme konularıyla ilgili hem teorik çalışmaların hem de pratik uygulamalar içeren çalışmaların sayısının ve yaygınlığının daha da artacağı öngörülmektedir. Bu nedenle çalışmamızın; doğal dil işleme alanında derin öğrenme uygulamaları konusunda önemli bir Türkçe kaynak olacağı düşünülmektedir.

Keywords

• Doğal Dil İşleme,Metin Madenciliği,Derin Öğrenme,Makine Öğrenmesi

References

1. Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning 2(1), 1-127.
2. Chen, Y., Xu, L., Lıu, K., Zeng, D., & Zhao, J. (2015). “Event extraction via dynamic multi-pooling convolutional neural networks”. Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing, 167-176.
3. Chiu, J. P., & Nıchols, E. (2015). “Named entity recognition with bidirectional LSTM-CNNs”. arXiv preprint arXiv:1511.08308.
4. Cho, Y., & Saul, L. K. (2009). “Kernel methods for deep learning”. Advances in Neural Information Processing Systems, 342-350.
5. Collobert, R. (2011). “Deep learning for efficient discriminative parsing”. International Conference on Artificial Intelligence and Statistics, 224-232.
6. Collobert, R., & Weston, J. (2008). “A unified architecture for natural language processing: Deep neural networks with multitask learning”. International Conference on Machine Learning (ICML), 160-167.
7. Conneau, A., Schwenk, H., Barrault, L., & Lecun, Y. (2016). “Very deep convolutional networks for natural language processing”. arXiv preprint arXiv:1606.01781.
8. Deng, L., & Yu, D. (2014). “Deep learning: methods and applications”. Foundations and Trends in Signal Processing, 7(3–4), 197-387.

9. Deselaers, T., Hasan, S., Bender, O., & Ney, H. (2009). “A deep learning approach to machine transliteration”. International Workshop on Statistical Machine Translation, 233-241.
10. Do, H. W., & Jeong, Y. S. (2016). “Temporal relation classification with deep neural network”. International Conference on Big Data and Smart Computing (BigComp), 454-457.
11. Dos Santos, C. N., & Gattı, M. (2014). “Deep convolutional neural networks for sentiment analysis of short texts”. International Conference on Computational Linguistics (COLING), 69-78.
12. Dos Santos, C. N., & Zadrozny, B. (2014). “Learning character-level representations for part-of-speech tagging”. International Conference on Machine Learning (ICML), 1818-1826.
13. Glorot, X., Bordes, A., & Bengıo, Y. (2011). “Domain adaptation for large-scale sentiment classification: A deep learning approach”. International Conference on Machine Learning (ICML), 513-520.
14. Hinton, G. E., & Salakhutdinov, R. R. (2006). “Reducing the dimensionality of data with neural networks”. Science, 313(5786), 504-507.
15. Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). “A fast learning algorithm for deep belief nets”. Neural Computation, 18(7), 1527-1554.
16. Hinton, G. E., & Sejnowski, T. J. (1986). “Learning and releaming in boltzmann machines”. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1(282-317), 2.
17. Hochreiter, S., & Schmidhuber, J. (1997). “Long short-term memory”. Neural Computation, 9(8), 1735-1780.
18. Kim, Y. (2014). “Convolutional neural networks for sentence classification”. arXiv preprint arXiv:1408.5882.
19. Lai, S., Xu, L., Liu, K., & Zhao, J. (2015). “Recurrent convolutional neural networks for text classification”. AAAI Conference on Artificial Intelligence, 2267-2273.
20. Lecun, Y., Bengio, Y., & Hinton, G. (2015). “Deep learning”. Nature 521(7553), 436.
21. Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). “Gradient-based learning applied to document recognition”. Proceedings of the IEEE, 86(11), 2278-2324.
22. Lee, D. H. (2013). “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks”. Workshop on Challenges in Representation Learning, ICML (3), 2
23. Lee, J. Y., & Dernoncourt, F. (2016). “Sequential short-text classification with recurrent and convolutional neural networks”. arXiv preprint arXiv:1603.03827.
24. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., & Ng, A. Y. (2011). “Multimodal deep learning”. International Conference on Machine Learning, 689-696.
25. Pang, B., & Lee, L. (2008). “Opinion mining and sentiment analysis”. Foundations and Trends in Information Retrieval, 2(1–2), 1-135.
26. Plank, B., Søgaard, A., & Goldberg, Y. (2016). “Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss”. arXiv preprint arXiv:1604.05529.
27. Qi, Y., Das, S. G., Collobert, R., & Weston, J. (2014). “Deep learning for character-based information extraction”. European Conference on Information Retrieval, 668-674.
28. Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G. Z. (2017). “Deep learning for health informatics”. IEEE journal of Biomedical and Health Informatics 21(1), 4-21.
29. Rosenblatt, F. (1958). “The perceptron: a probabilistic model for information storage and organization in the brain”. Psychological Review, 65(6), 386.
30. Salakhutdinov, R., & Larochelle, H. (2010). “Efficient learning of deep Boltzmann machines”. International Conference on Artificial Intelligence and Statistics.
31. Severyn, A., & Moschitti, A. (2015) a. “Twitter sentiment analysis with deep convolutional neural networks”. International ACM SIGIR Conference on Research and Development in Information Retrieval, 959-962.
32. Severyn, A., & Moschitti, A. (2015) b. “Learning to rank short text pairs with convolutional deep neural networks”. International ACM SIGIR Conference on Research and Development in Information Retrieval, 373-382.
33. Socher, R., Bengio, Y., & Manning, C. D. (2012). “Deep learning for NLP (without magic)”. Tutorial Abstracts of ACL 2012, 5.
34. Tan, M., Santos, C. D., Xiang, B., & Zhou, B. (2015). “LSTM-based deep learning models for non-factoid answer selection”. arXiv preprint arXiv:1511.04108.
35. Williams, R. J., & Zipser, D. (1989). “A learning algorithm for continually running fully recurrent neural networks”. Neural Computation, 1(2), 270-280.
36. Yu, L., Hermann, K. M., Blunsom, P., & Pulman, S. (2014). “Deep learning for answer sentence selection”. arXiv preprint arXiv:1412.1632.
37. Zhai, S., & Zhang M. Z. (2016). “Semisupervised autoencoder for sentiment analysis”. AAAI Conference on Artificial Intelligence, 1394-1400.
38. Zhang, X., Zhao, J., & Lecun, Y. (2015). “Character-level convolutional networks for text classification”. Advances in Neural Information Processing Systems, 649-657.