A Comparison of LSTM and BERT Based Models in Turkish Keyword Extraction

Year 2024, , 9 - 18, 11.06.2024

Özlem Aydın , Hüsein Kantarcı

https://doi.org/10.54525/bbmd.1454220

Abstract

Nowadays, text-based data on the internet is increasing very rapidly and it is an important need to reach the right content that cntains the desired information from this big data. Knowing the keywords of the content can provide a positive effect in meeting this need. In this study, it is aimed to determine the keywords representing Turkish texts with natural language processing and deep learning models. Turkish Labeled Text Corpus and Text Summarization- Keyword Extraction Dataset were used together as dataset. Two different deep learning models were presented in this study. Firstly, Sequence-to-Sequence (Seq2Seq) Model with Long Short-Term Memory (LSTM) layers is designed. The other model is a Seq2Seq model with BERT (Bidirectional Encoder Representations from Transformers). In the evaluation of success of the LSTM layered Seq2seq model, an F-1 score of 0.38 was achieved in the ROUGE-1 criterion. In the BERT-based Seq2Seq model, an F-1 value of 0.399 was obtained in the ROUGE-1 criterion. As a result, it has been observed that the BERT based Seq2Seq model based on the Transformer architecture is more successful than the LSTM based Seq2Seq model.

Keywords

Keyword extaction, Deep learning, Seq2seq architecture, Transformer architecture

Türkçe Anahtar Sözcük Çıkarımında LSTM ve BERT Tabanlı Modellerin Karşılaştırılması

Abstract

Günümüzde internet ortamında metne dayalı veri çok hızlı bir şekilde artış göstermektedir ve bu büyük veri içinden istenilen bilgiyi barındıran doğru içeriklere ulaşabilmek önemli bir ihtiyaçtır. İçeriklere ait anahtar sözcüklerin bilinmesi bu ihtiyacı karşılamada olumlu bir etki sağlayabilmektedir. Bu çalışmada, doğal dil işleme ve derin öğrenme modelleri ile Türkçe metinleri temsil eden anahtar sözcüklerin belirlenmesi amaçlanmıştır. Veri kümesi olarak Türkçe Etiketli Metin Derlemi ve Metin Özetleme-Anahtar Kelime Çıkarma Veri Kümesi birlikte kullanılmıştır. Derin öğrenme modeli olarak çalışmada iki farklı model ortaya konmuştur. İlk olarak Uzun Ömürlü Kısa Dönem Belleği ( LSTM) katmanlı bir Diziden Diziye (Seq2Seq) model tasarlanmıştır. Diğer model ise BERT (Transformatörler ile İki Yönlü Kodlayıcı Temsilleri) ile oluşturulmuş Seq2Seq bir modeldir. LSTM katmanlı Seq2seq modelin başarı değerlendirmesinde ROUGE-1 ölçütünde 0,38 F-1 değerine ulaşılmıştır. BERT tabanlı Seq2Seq modelde ROUGE-1 ölçütünde 0,399 F-1 değeri elde edilmiştir. Sonuç olarak dönüştürücü mimarisini temel alan BERT tabanlı Seq2Seq modelin, LSTM tabanlı Seq2seq modele görece daha başarılı olduğu gözlemlenmiştir.

Keywords

Anahtar sözcük çıkarımı, Derin öğrenme, Seq2Seq mimarisi, Transformatör mimarisi

References

• Hashemzahde, B. Ve ark., Improving keyword extraction in multilingual texts, Int J Electric Comput Eng, 2020, 10:5909-5916.
• Papagiannopoulou, E., Tsoumakas, G., A review of keyphrase extraction, CoRR, 2019.
• Witten, I. H., Paynter, G. W., Frank E., Gutwin, C., NevillManning, C. G., Kea: Practical Automatic Keyphrase Extraction, In Proceedings of the 4th ACM Conf. of the Digital Libraries, 1999, Berkeley, CA, USA.
• Turney, P., Learning algorithms for keyphrase extraction, Information Retrieval, 2000, 2:303–336.
• Zhang, Q., Wang, Y., Gong, Y., Keyphrase extraction using deep recurrent neural networks on Twitter, In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2016, Austin, Texas, pp. 836–845.
• Zhang, Y., Yang, F., Xiao, W., Deep keyphrase generation with a convolutional sequence to sequence model, In Proceedings of the 4th International Conference on Systems and Informatics, Hangzhou, 2017, China, pp. 1477–1485.
• Chen, W., Gao, Y., Zhang, J., King, I., Lyu, M. R., Title-guided encoding for keyphrase generation, In Proceedings of AAAI Conference on Artificial Intelligence, 2019, pp. 6268–6275.
• Mihalcea, R., Tarau, P., TextRank: Bringing order into text, Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP '04), 2004, Barcelona, Spain, pp. 404-411.
• Wan, X., Xiao, J., Single document keyphrase extraction using neighborhood knowledge, In Proceedings of the 23rd AAAI Conference on Artificial Intelligence, 2008, pp. 855-860.
• Gollapalli, S. D., Caragea, C., Extracting keyphrases from research papers using citation networks, In Proceedings of the 28th AAAI Conference on Artificial Intelligence, 2014, pp. 1629-1635.
• Rose, S., Engel, D., Cramer, N., Cowley, W., Automatic keyword extraction from individual documents, Text Mining:Applications and Theory, 2010, pp. 1-20.
• Liu, Z., Li, P., Zheng, Y., Sun, M., Clustering to find exemplar terms for keyphrase extraction, In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 2009, Singapore, pp. 257–266.
• Bougouin, A., Boudin, F., Daille, B., TopicRank: Graph-based topic ranking for keyphrase extraction, In Proceedings of the 6th International Joint Conference on Natural Language Processing, 2013, pp. 543-551.
• Zha, H., Generic summarization and keyphrase extraction using mutual reinforcement principle and sentence clustering, In Proceedings of 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2002, pp. 113-120.
• Tomokiyo, T., Hurst, M., A language model approach to keyphrase extraction, In Proceedings of the ACL 2003Workshop on Multiword Expressions: Analysis, Acquisition and Treatment - Volume 18. Association for Computational Linguistics MWE’03, 2003, pp. 33-40.
• Pala, N., Cicekli, I., Turkish keyphrase extraction using kea, In Proceedings of the 22nd International Symposium on Computer and Information Sciences (ISCIS 2007), 2007, Ankara, pp. 1-5.
• Kalaycilar, F., Cicekli, I., TurKeyx: Turkish keyphrase extractor, In Proceedings of the 23rd International Symposium on Computer and Information Sciences (ISCIS 2008), 2008, Istanbul, pp. 1-4.
• Ozdemir, B., Cicekli, I., Turkish Keyphrase Extraction Using Multi-Criterion Ranking, In: 24th International Symposium on Computer and Information Sciences, 2009, pp. 269-273.
• Müngen, A. A., Kaya, M., Extracting abstract and keywords from context for academic articles, Social Network Analysis and Mining 8, 2018, pp. 1-11‏.
• Yıldız, O., Metin Madenciliğinde Anahtar Kelime Seçimi Bir Üniversite Örneği, Yönetim Bilişim Sistemleri Dergisi, 2(3), 2017, pp. 29-50.
• Ayan, E. T., Arslan, R., Zengin, M. S., Duru, H. A., Salman, S., Bardak, B., Turkish Keyphrase Extraction from Web Pages with BERT, In 29th Signal Processing and Communications Applications Conference, 2021.
• Göz, F., Mutlu, A., Küçük, K., Temur, M., Gün, A., Effect of Centrality Measures for Keyword Extraction from Turkish Documents, In 29th Signal Processing and Communications Applications Conference, 2021.
• Yıldız, A. M., Keyword Extraction with Textrank and Tfidf From Turkish Articles, The International Informatics Congress 2022 (IIC2022), 2022.
• Erzurumlu, H. Y., Akgul, Y. S., Adaptive Keyword Extraction Service for Turkish, IntelliSys (2) 2022, 2022, pp. 495-506.
• Natural Language Toolkit, https://www.nltk.org
• NumPy Kütüphanesi, https://numpy.org
• Pandas Kütüphanesi, https://pandas.pydata.org
• Tensorflow Kütüphanesi, https://www.tensorflow.org
• Rouge Scorer Kütüphanesi, https://pypi.org/project/rouge-scorer
• Öztürk, S., Sankur, B., Güngör, T., Yılmaz, M. B., Köroğlu, B., Ağın, O., İşbilen, M., Ulaş, Ç., Ahat, M., Türkçe Etiketli Metin Derlemi, 22. IEEE Sinyal İşleme ve İletişim Uygulamaları Kurultayı (SIU-14), 2014.
• Metin Özetleme-Anahtar Kelime Çıkarma Veri Seti, http://buyukveri.firat.edu.tr/veri-setleri
• Akın, Ahmet A., Zemberek-NLP, Github. https://github.com/ahmetaa/zemberek-nlp
• Sutskever, I., Vinyals, O., Le, Q. V., Sequence to sequence learning with neural networks, Proceedings of NeurIPS, 2014, pp. 3104-3112.
• Keras Kütüphanesi, https://keras.io
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., Attention is All You Need, In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), Long Beach-California-A.B.D, 6000-6010, 4-9 Aralık, 2017.
• Unzueta, D., Transformers: An Overview of the Most Novel AI Architecture,2022. https://towardsdatascience.com/transformers-an-overview-of-the-most-novel-ai-architecture
• Devlin, J., Chang, M.-W., Lee, K. ve Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1, 2019, pp. 4171-4186.
• Horev, R., BERT Explained: State of the art language model for NLP, 2018. https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270
• Lin C.-Y., ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out, Association for Computational Linguistics, 2004, pp. 74-81.