New Summary Text Generation from Long Texts With BiLSTM Deep Learning Method

Year 2024, , 1096 - 1119, 15.09.2024

Onur Şahin , Rıdvan Yayla

https://doi.org/10.31466/kfbd.1423022

Abstract

Nowadays, the integration of deep learning techniques into creative text generation processes is widely used among researchers and software developers. Deep learning is known for its ability to learn complex information over large data sets, and this feature offers significant advantages in language understanding and text generation. The aim of the study is to derive a new summary text by training the expressions in a long text on the basis of Bidirectional Long Short-Term Memory (BiLSTM) deep learning architecture. For this purpose, two documents of different lengths and types (Novel, Personal Development) written in Turkish were used as a dataset, and the texts in the dataset were subjected to a series of pre-processes such as data cleaning, tokenization and vectorization. The study evaluated other deep learning architectures such as LSTM, GRU, BiGRU and CNN, as well as BiLSTM, and found that the BiLSTM model had the highest METEOR, BLEU and ROGUE scores in two different book types and different word counts (1,000, 2,000 and 5,000 words). showed. These findings show that BiLSTM produces more successful results than other models for text summarization and text generation. The method of generating creative and original texts from a certain novel or personal development book using BiLSTM is an inspiring resource for researchers and software developers, and it is envisaged that the proposed method can be applied for different text types. In this way, it has been shown that the BiLSTM architecture produces successful results in text summarization and generation processes.

Keywords

BiLSTM, Natural language processing, Text generation, Deep learning

BiLSTM Derin Öğrenme Yöntemi ile Uzun Metinlerden Yeni Özet Metinlerin Türetilmesi

Abstract

Günümüzde, derin öğrenme tekniklerinin yaratıcı metin oluşturma süreçlerine entegrasyonu, araştırmacılar ve yazılım geliştiriciler arasında yaygın olarak kullanılmaktadır. Derin öğrenme, büyük veri setleri üzerinde karmaşık bilgileri öğrenme yeteneği ile bilinir ve bu özellik, dil anlama ve metin üretme konularında önemli avantajlar sunar. Çalışmanın amacı Bidirectional Long Short-Term Memory (BiLSTM) derin öğrenme mimarisi temelinde uzun bir metindeki ifadelerin eğitilerek yeni bir özet metnin türetilmesidir. Bu amaç doğrultusunda Türkçe dilinde yazılmış farklı uzunlukta ve türdeki (Roman, Kişisel Gelişim) iki doküman veriseti olarak kullanılmış, veri setindeki metinler veri temizleme, tokenizasyon ve vektörleştirme gibi bir dizi önişlemden geçirilmiştir. Çalışma, BiLSTM'nin yanı sıra LSTM, GRU, BiGRU ve CNN gibi diğer derin öğrenme mimarilerini de değerlendirmiş ve BiLSTM modelinin iki farklı kitap türünde ve farklı kelime sayılarında (1.000, 2.000 ve 5.000 kelime) en yüksek METEOR, BLEU ve ROGUE skorlarına sahip olduğunu ortaya koymuştur. Bu bulgular, BiLSTM’nin metin özetleme ve metin üretme için diğer modellere göre daha başarılı sonuçlar ürettiğini göstermektedir. BiLSTM kullanarak belli bir roman veya kişisel gelişim kitabından yaratıcı ve özgün metinler türetme yöntemi araştırmacılar ve yazılım geliştiriciler için ilham verici bir kaynak olup, önerilen yöntemin farklı metin türleri için de uygulanabileceği öngörülmektedir. Bu sayede, metin özetleme ve üretme süreçlerinde BiLSTM mimarisinin başarılı sonuçlar ürettiği gösterilmiştir.

Keywords

BiLSTM, Doğal dil işleme, Metin üretme, Derin öğrenme

Ethical Statement

Yapılan çalışmada araştırma ve yayın etiğine uyulmuştur.

References

• Babüroğlu, B., Tekerek, A., & Tekerek, M. (2019). Türkçe İçin Derin Öğrenme Tabanlı Doğal Dil İşleme Modeli Geliştirilmesi. 13th International Computer and Instructional Technology Symposium
• Banerjee, S. and Lavie, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
• Bayer, M., Kaufhold, M.-A., Buchhold, B., Keller, M., Dallmeyer, J., Reuter, C. (2022). Data augmentation in natural language processing: a novel text generation approach for long and short text classifiers. International Journal of Machine Learning and Cybernetics, 14(3), 135-150. https://doi.org/10.1007/s13042-022-01553-3
• Chakraborty, S., Banik, J., Addhya, S., & Chatterjee, D. (2020). Study of Dependency on number of LSTM units for Character based Text Generation models. 2020 International Conference on Computer Science, Engineering and Applications (ICCSEA).
• Cui, P., Wang, X., Pei, J., & Zhu, W. (2018). A Survey on Network Embedding. IEEE Transactions on Knowledge and Data Engineering, 31(5), 833-852. https://doi.org/10.1109/TKDE.2018.2849727
• Dey, R., Salem F. M. (2017). Gate-variants of Gated Recurrent Unit (GRU) neural networks. Midwest Symposium on Circuits and Systems, 1597-1600. https://doi.org/10.1109/MWSCAS.2017.8053243
• Erhandi, B., Çallı, F. (2020). Derin Özetleme ile Metin Özetleme. 3rd International Conference on Data Science and Applications (ICONDATA’20).
• Fagin, R., Kumar, R., Sivakumar, D. (2003). Comparing Top k Lists. SIAM Journal on Discrete Mathematics 17(1), 134-160. https://doi.org/10.1137/S0895480102412856
• Fang, T., Jaggi, M., & Argyraki, K. (2017). Generating Steganographic Text with LSTMs. https://doi.org/10.48550/arXiv.1705.10742
• Fatima, N., Imran, A. S., Kastrati, Z., Daudpota, S. M., & Soomro, A. (2021). A Systematic Literature Review on Text Generation Using Deep Neural Network Models. 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).
• Grusky, M. (2023, July). Rogue scores. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1914-1934).
• Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., & Chen, T. (2018). Recent advances in convolutional neural networks. Pattern Recognition, 77, 354-377. https://doi.org/10.1016/j.patcog.2017.10.013
• Guo, B., Wang, H., Ding, Y., Wu, W., Hao, S., Sun, Y., & Yu, Z. (2020). Conditional Text Generation for Harmonious Human-Machine Interaction. ACM Transactions on Intelligent Systems and Technology 14, 1–50. https://doi.org/10.1145/3439816
• Huang, Z., Xu, W., Yu, K. (2015). Bidirectional LSTM-CRF Models for Sequence Tagging. https://doi.org/10.48550/arXiv.1508.01991
• Iqbal, T., Qureshi, S. (2020). The survey: Text generation models in deep learning. Journal of King Saud University - Computer and Information Sciences 34/6, 2515-2528. https://doi.org/10.1016/j.jksuci.2020.04.001
• Javid, A. M., Das, S., Skoglund, M., & Chatterjee, S. (2021). A ReLU Dense Layer to Improve the Performance of Neural Networks. ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2810-2814.
• Kandel, I., Castelli, M. (2020). The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset. ICT Express, 6(4), 312-315. https://doi.org/10.1016/J.ICTE.2020.04.010
• Li, J., Zhao, W. X., Nie, J.-Y., Wen, J.-R., & Tang, T. (2022). Pre-trained Language Models for Text Generation: A Survey. https://doi.org/10.48550/arXiv.2201.05273
• Li, L., Zhang, T. (2021). Research on Text Generation Based on LSTM. International Core Journal of Engineering, 7, 2021. https://doi.org/10.6919/ICJE.202105_7(5).0067
• Li, X., Ma, X., Xiao, F., Xiao, C., Wang, F., & Zhang, S. (2022). Time-series production forecasting method based on the integration of Bidirectional Gated Recurrent Unit (Bi-GRU) network and Sparrow Search Algorithm (SSA). Journal of Petroleum Science and Engineering, 208, 109309. https://doi.org/10.1016/J.PETROL.2021.109309
• Lin, C. Y. (2004, July). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out (pp. 74-81).
• Mao, Y., Li, X., Li, Z., & Li, W. (2024). Automated Smart Contract Summarization via LLMs. arXiv preprint arXiv:2402.04863.
• Mishra, R., Bian, J., Fiszman, M., Weir, C. R., Jonnalagadda, S., Mostafa, J., & Fiol, G. Del. (2014). Text summarization in the biomedical domain: A systematic review of recent research. Journal of Biomedical Informatics. 52, 457-467 https://doi.org/10.1016/j.jbi.2014.06.009
• Onan, A. (2022). Türkçe Metin Madenciliği için Çalışan Bellek Bağlantıları Tabanlı Uzun Kısa Süreli Bellek Mimarisi. European Journal of Science and Technology Special Issue, 34, 239-246. https://doi.org/10.31590/ejosat.1080239
• Onan, A. (2022). Türkçe Metin Madenciliği için Dikkat Mekanizması Tabanlı Derin Öğrenme Mimarilerinin Değerlendirilmesi. European Journal of Science and Technology Special Issue, 34, 403-407. https://doi.org/10.31590/ejosat.1082379
• Otter, D. W., Medina, J. R., & Kalita, J. K. (2021). A Survey of the Usages of Deep Learning for Natural Language Processing. IEEE Transactions on Neural Networks and Learning Systems. Issue, 2, 604-624.
• Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002, July). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311-318).
• Pawade, D., Sakhapara, A., Somaiya, K. J., Jain, M., Jain, N., & Gada, K. (2018). Story Scrambler - Automatic Text Generation Using Word Level RNN-LSTM. Information Technology and Computer Science, 6, 44-53. https://doi.org/10.5815/ijitcs.2018.06.05
• Samant, R. M., Bachute, M. R., Gite, S., & Kotecha, K. (2022). Framework for Deep Learning-Based Language Models Using Multi-Task Learning in Natural Language Understanding: A Systematic Literature Review and Future Directions.
• Santhanam, S. (2020). Context Based Text - Generation Using LSTM Networks. https://doi.org/10.48550/arXiv.2005.00048
• Semeniuta, S., Severyn, A., & Barth, E. (2017). A Hybrid Convolutional Variational Autoencoder for Text Generation. https://doi.org/10.48550/arXiv.1702.02390
• Sherstinsky, A. (2020). Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306. https://doi.org/10.1016/J.PHYSD.2019.132306
• Shi, Z., Chen, X., Qiu, X., & Huang, X. (2018). Toward Diverse Text Generation with Inverse Reinforcement Learning. https://doi.org/10.48550/arXiv.1804.11258
• Sutskever, I., Martens, J., & Hinton, G. (2011). Generating Text with Recurrent Neural Networks.
• Ünlü, Ö., Cetin, A. (2019). A Survey on Keyword and Key Phrase Extraction with Deep Learning. 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies, ISMSIT 2019 - Proceedings.
• Vo, S. N., Vo, T. T., & Le, B. (2024). Interpretable extractive text summarization with meta-learning and BI-LSTM: A study of meta learning and explainability techniques. Expert Systems with Applications, 245, 123045.
• Wang, M., Lu, S., Zhu, D., Lin, J., & Wang, Z. (2018). A High-Speed and Low-Complexity Architecture for Softmax Function in Deep Learning. 2018 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2018, 223-226. https://doi.org/10.1109/APCCAS.2018.8605654
• Welleck, S., Brantley, K., Daumé, H., & Cho, K. (2019). Non-Monotonic Sequential Text Generation. Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6716-6726.
• Wu, N., Gong, M., Shou, L., Liang, S., & Jiang, D. (2023, October). Large language models are diverse role-players for summarization evaluation. In CCF International Conference on Natural Language Processing and Chinese Computing (pp. 695-707). Cham: Springer Nature Switzerland.
• Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent Trends in Deep Learning Based Natural Language Processing [Review Article]. IEEE Computational Intelligence Magazine. 13, 55-75.
• Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022). A Survey of Knowledge-Enhanced Text ACM Computing Surveys. 227, 1-38. https://doi.org/10.1145/3512467
• Zhang, T., Meng, J., Yang, Y., & Yu, S. (2024). Contrastive learning penalized cross-entropy with diversity contrastive search decoding for diagnostic report generation of reduced token repetition. Applied Sciences, 14(7), 2817.
• Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., ... & Wen, J. R. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223.
• Zhou, W., Ye, Z., Yang, Y., Wang, S., Huang, H., Wang, R., & Yang, D. (2023). Transferring pre-trained large language-image model for medical image captioning. In CLEF2023 Working Notes, CEUR Workshop Proceedings, CEUR-WS. org, Thessaloniki, Greece.
• Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., Yu, Y. (2018). Texygen: A Benchmarking Platform for Text Generation Models. SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval 1097–1100. https://doi.org/10.1145/3209978.3210080