Anlatım Bozukluklarının Tespit Edilmesi için Hiperparametre Optimizasyon Teknikleri Kullanılarak Derin Öğrenme Yöntemlerinin Potansiyelinin Artırılması

Abstract

Doğal Dil İşleme (DDİ), derin öğrenme çalışmaları alanında dikkat çekici ilerlemeler ortaya koymuştur. Derin öğrenme tabanlı teknikler, yalnızca kural tabanlı DDİ yöntemlerine üstün bir alternatif olmakla kalmayıp, aynı zamanda metin sınıflandırma, duygu analizi veya belge kümeleme gibi çeşitli DDİ görevlerinde de daha doğru performanslar elde etmeyi başarmıştır. Bir derin öğrenme modelinin performansı, şüphesiz ki hiperparametrelerinin ideal şekilde ayarlanmasına bağlı olduğundan, en ideal hiperparametrelerin ayarlanması, girdi verilerinden anlamlı örüntü çıkarma açısından model öğrenmesinin kapasitesini belirler. Bu makalede, Türkçe cümlelerdeki anlatım bozukluklarını tespit etmek amacıyla Uzun Kısa-Süreli Bellek (UKSB) ve Evrişimsel Sinir Ağları (ESA) derin öğrenme modelleri üzerinde Bayesian Optimization, Random Search ve Grid Search hiperparametre optimizasyon teknikleri uygulanmıştır. Bu amaçla daha önce geliştirilmiş UKSB ve ESA modellerinin hiperparametreleri, zaman alan ve genel olarak en ideal modeli garanti edemeyen deneme-yanılma yaklaşımı kullanılarak ayarlanmıştı. Bu hiperparametreler, optimizasyon teknikleri kullanılarak ayarlandıktan sonra ise, doğruluk açısından performansları UKSB ve ESA modelleri için sırasıyla %87,94'ten %92,82'ye ve %84,33'ten %89,79'a yükseltilmiştir.

Keywords

• Bayesian optimization
• Grid search
• Hiperparametre optimizasyonu
• Doğal dil işleme
• Random search
• Türkçe

Unleashing the Potential of Deep Learning Methods for Detecting Defective Expressions Using Hyperparameter Optimization Techniques

Abstract

Natural Language Processing (NLP) has emerged remarkable progress in the field of deep learning studies. Not only a superior alternative to rule-based NLP methods, deep learning-based techniques have also succeeded more accurate performances in various NLP tasks such as text classification, sentiment analysis or document clustering. Since the performance of a deep learning model undoubtedly depends on adjusting its hyperparameters ideally, tuning the most optimum hyperparameters determines the capability of the model learning in terms of meaningful pattern extraction from the input data. In this paper, hyperparameter optimization techniques of Bayesian Optimization, Random Search and Grid Search have been applied on the deep learning models of Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) for the purpose of detecting defective expressions in Turkish sentences. The hyperparameters of previously implemented LSTM and CNN models for this purpose have been adjusted using trial-and-error approach, which is time-consuming and cannot guarantee the most ideal model in general. After these hyperparameters have been adjusted using optimization techniques, the performances in terms of accuracy have been increased from 87.94% to 92.82% and from 84.33% to 89.79% for the models of LSTM and CNN respectively.

Keywords

• Bayesian optimization
• Grid search
• Hyperparameter optimization
• NLP
• Random search
• Turkish

References

1. [1] LeCun, Y., Bengio, Y., & Hinton, G., 2015. Deep learning. Nature, Vol. 521(7553), pp. 436-444.
2. [2] Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R., 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580.
3. [3] Bergstra, J., Bengio, Y., 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, Vol. 13(2).
4. [4] Smith, L.N., 2018. A disciplined approach to neural network hyper-parameters: Part 1--learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820.
5. [5] Lilhore, U.K., Dalal, S., Faujdar, N., Margala, M., Chakrabarti, P., Chakrabarti, T., Velmurugan, H., 2023. Hybrid CNN-LSTM model with efficient hyperparameter tuning for prediction of Parkinson’s disease. Scientific Reports, Vol. 13(1), p. 14605.
6. [6] El Ghazi, M., Aknin, N., 2024. Optimizing Deep LSTM Model through Hyperparameter Tuning for Sensor-Based Human Activity Recognition in Smart Home. Informatica, Vol. 47(10).
7. [7] Mehta, R., Jurečková, O., Stamp, M., 2024. A natural language processing approach to Malware classification. Journal of Computer Virology and Hacking Techniques, Vol. 20(1), pp. 173-184.
8. [8] Palaniammal, M.A., Anandababu, P., 2024. Enhancing Sarcasm Recognition Using Chicken Swarm Optimization Algorithm with Graph Neural Network on Social Media.

9. [9] Demir, C., 2020. Lexical and structural ambiguities in student writing: An assessment and evaluation of results. Academic Education Research Journal, Vol. 8, pp. 100-108. DOI: 10.30918/AERJ.8S3.20.077.
10. [10] Göksel, A., Kerslake, C., 2004. Turkish: A comprehensive grammar. Routledge.
11. [11] Büyükikiz, K.K., 2007. İlköğretim 8. sınıf öğrencilerinin yazılı anlatım becerilerinin söz dizimi ve anlatım bozukluğu açısından değerlendirilmesi. Gazi University, Ankara, Turkey.
12. [12] Özdem, A., 2012. Çanakkale'deki yerel gazetelerin anlatım bozuklukları açısından incelenmesi. Çanakkale Onsekiz Mart University, Çanakkale, Turkey.
13. [13] Suncak, A., Aktaş, Ö., 2021. A novel approach for detecting defective expressions in Turkish. Journal of Artificial Intelligence and Data Science (JAIDA), Vol. 1, pp. 35-40.
14. [14] Suncak, A., 2022. Developing a new approach in natural language understanding to defect defective expressions in Turkish sentences. Dokuz Eylül University, İzmir, Turkey.
15. [15] Suncak, A., Aktaş, Ö., 2022. Detecting Defective Expressions in Turkish Sentences Using a Hybrid Deep Learning Method. Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, Vol. 24(72), pp. 825-834.
16. [16] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B., 2011. Algorithms for hyper-parameter optimization. Advances in Neural Information Processing Systems, Vol. 24.
17. [17] Chollet, F., 2015. Keras. GitHub. Retrieved from https://github.com/fchollet/keras.
18. [18] Abadi, M., et al., 2016. TensorFlow. GitHub. Retrieved from https://github.com/tensorflow.
19. [19] Aktaş, Ö., Birant, Ç.C., Aksu, B., Çebi, Y., 2013. Automated synonym dictionary generation tool for Turkish (ASDICT). Bilig, Vol. 65, p. 47.
20. [20] Muhammad, P.F., Kusumaningrum, R., Wibowo, A., 2021. Sentiment analysis using Word2vec and long short-term memory (LSTM) for Indonesian hotel reviews. Procedia Computer Science, Vol. 179, pp. 728-735.
21. [21] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J., 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, pp. 3111-3119.
22. [22] Fang, G., Zeng, F., Li, X., Yao, L., 2021. Word2vec based deep learning network for DNA N4-methylcytosine sites identification. Procedia Computer Science, Vol. 187, pp. 270-277.
23. [23] Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural Computation, Vol. 9(8), pp. 1735-1780.
24. [24] Gers, F.A., Schmidhuber, J., Cummins, F., 2000. Learning to forget: Continual prediction with LSTM. Neural Computation, Vol. 12(10), pp. 2451-2471.
25. [25] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, Vol. 86(11), pp. 2278-2324.
26. [26] Hossin, M., Sulaiman, M.N., 2015. A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process, Vol. 5(2), p. 1.
27. [27] Feurer, M., Hutter, F., 2019. Hyperparameter optimization. Automated Machine Learning: Methods, Systems, Challenges, pp. 3-33.
28. [28] Michalski, R.S., Carbonell, J.G., Mitchell, T.M., 1984. Machine learning: An artificial intelligence approach. Springer-Verlag Berlin Heidelberg.
29. [29] Pontes, F.J., Amorim, G.F., Balestrassi, P.P., Paiva, A.P., Ferreira, J.R., 2016. Design of experiments and focused grid search for neural network parameter optimization. Neurocomputing, Vol. 186, pp. 22-34.
30. [30] Ensor, K.B., Glynn, P.W., 1997. Stochastic optimization via grid search. Lectures in Applied Mathematics, Vol. 33, pp. 89-100.
31. [31] Andradóttir, S., 2006. An overview of simulation optimization via random search. Handbooks in Operations Research and Management Science, Vol. 13, pp. 617-631.
32. [32] Andradóttir, S., 2014. A review of random search methods. Handbook of Simulation Optimization, pp. 277-292.
33. [33] Močkus, J., 1975. On Bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference, pp. 400-404.
34. [34] Mockus, J., Mockus, J., 1989. The Bayesian approach to local optimization. Springer Netherlands, pp. 125-156.
35. [35] Frazier, P.I., 2018. Bayesian optimization. In Recent Advances in Optimization and Modeling of Contemporary Problems, pp. 255-278.
36. [36] Snoek, J., Larochelle, H., Adams, R.P., 2012. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, Vol. 25.