Ön eğitimli Bert modeli ile patent sınıflandırılması

Yıl 2024, Cilt: 39 Sayı: 4, 2484 - 2496, 20.05.2024

Selen Yücesoy Kahraman , Alptekin Durmuşoğlu , Türkay Dereli

https://doi.org/10.17341/gazimmfd.1292543

Öz

Patentler, bilgi teknolojilerindeki yeniliklerin korunmasına yardımcı olan ve bu yeniliklerin yaratıcısına belirli bir süre boyunca özel haklar sağlayan belgelerdir. Bu haklar, patent sahibine yeniliği ticari olarak kullanma hakkı verirken, başkalarının yeniliği izinsiz kullanmasını engeller. Radikal yenilikler ve çığır açan teknolojik gelişmeler, mevcut patentlerde yer alan teknik bilgilerden türetilmiştir. Otomatik bir sınıflandırma sistemi kullanılarak, ait oldukları teknik sınıfa atanan patentler, araştırmacıların önünü açabilmekte ve yeni buluşlar yaratabilecekleri bir ortam sağlayabilmektedir. Bu çalışma, BERT algoritmasını kullanarak otomatik bir patent sınıflandırma analizi sunmaktadır. Otomatik patent sınıflandırma problemlerinde daha başarılı tahmin doğruluğuna ulaşabilmek için yapılan hiper parametre analizleri bu çalışmada da tercih edilmiştir. Elde edilen sonuçlar literatürdeki sonuçlarla rekabet edecek düzeydedir. Bu çalışmada alt sınıf düzeyinde % 55,9 tahmin doğruluğu elde edilmiştir.

Anahtar Kelimeler

Patent classification, Bert, deep learning, text classification

Destekleyen Kurum

Destekleyen bir kurum bulunmamaktadır.

Proje Numarası

Proje numarası bulunmamaktadır.

Kaynakça

• 1. Asche, G., “80% of technical information found only in patents” – Is there proof of this ?, World Patent Information, 48, 16–28, 2017.
• 2. Seo, W. A patent-based approach to identifying potential technology opportunities realizable from a firm’s internal capabilities, Computers & Industrial Engineering (Comput. Ind. Eng), 171, 108395, 2022.
• 3. Sofean, M., Deep learning based pipeline with multichannel inputs for patent classification, World Patent Information, 66, 102060, 2021.
• 4. Giordano V., Chiarello F., Melluso N., Fantoni G. and Bonaccorsi A., Text and Dynamic Network Analysis for Measuring Technological Convergence: A Case Study on Defense Patent Data, IEEE Transactions on Engineering Management (IEEE Trans. Eng. Manage), 70, 4, 1490-1503, 2023.
• 5. Durmusoglu, A., Updating technology forecasting models using statistical control charts, Kybernetes, 47 (4), 672–688, 2018.
• 6. Hain, D.S., Jurowetzki, R., Buchmann, T., and Wolf, P. A., Text-Embedding-based Approach to Measure Patent-to-Patent Technological Similarity, Technological Forecasting and Social Change (Technol. Forecasting Social Change), 177, 121559, 2022.
• 7. Yan, B., and Luo, J., Measuring technological distance for patent mapping. The Journal of the Association for Information Science and Technology, 68 (2), 423–437, 2017.
• 8. Erzurumlu, S.S., and Pachamanova, D., Topic modeling and technology forecasting for assessing the commercial viability of healthcare innovations. Technological Forecasting and Social Change (Technol. Forecasting Social Change), 156, 120041, 2020.
• 9. Grawe, M.F., Martins, C.A., and Bonfante, A.G., Automated patent classification using word embedding, Proceedings - 16th IEEE International Conference on Machine Learning and Applications, ICMLA 2017, Cancun, Mexico, 408–411, 18-21 Aralık 2017.
• 10. Altuntas, S., Dereli, T., and Kusiak, A., Analysis of patent documents with weighted association rules, Technological Forecasting and Social Change (Technol. Forecasting Social Change), 92, 249–262, 2015.
• 11. Mutlu N.G., Altuntaş S., Monitoring technological changes with statistical control charts based on patent data, Journal of the Faculty of Engineering and Architecture of Gazi University, 36 (4), 1875-1892, 2021.
• 12. Li, S., Hu, J., Cui, Y., and Hu, J., DeepPatent: patent classification with convolutional neural networks and word embedding, Scientometrics, 117, 721–744, 2018.
• 13. Yücesoy Kahraman, S., Dereli, T., Durmuşoglu, A. Forty years of automated patent classification, International Journal of Information Technology & Decision Making (Int. J. Inf. Technol. Decis. Making), 2023. https://doi.org/10.1142/S0219622023500165
• 14. Fall, C.J., Törcsvári, A., Fiévet, P., and Karetka, G., Automated categorization of German-language patent documents, Expert Systems with Applications (Expert Syst. Appl), 26 (2), 269–277, 2004.
• 15. Fall, C., and Benzineb, K., Literature survey: Issues to be considered in the automatic classification of patents, World Intellectual Property Organization, 2002.
• 16. Piroi, F., and Hanbury, A., Multilingual Patent Text Retrieval Evaluation: CLEF–IP, Information Retrieval Evaluation in a Changing World, The Information Retrieval Series, Springer Nature Switzerland AG, 41, 365–387, 2019.
• 17. Roda, G., Tait, J., Piroi, F., and Zenz, V., CLEF-IP 2009: Retrieval Experiments in the Intellectual Property Domain, Multilingual Information Access Evaluation I. Text Retrieval Experiments CLEF 2009, Lecture Notes in Computer Science, 6241. Springer, Berlin, Heidelberg, 2010.
• 18. Karaca A., Aydın Ö., Generating headlines for Turkish news texts with transformer architecture based deep learning method, Journal of the Faculty of Engineering and Architecture of Gazi University, 39 (1), 485–496, 2023.
• 19. Lee, J.S., and Hsiang, J., Patent classification by fine-tuning BERT language model, World Patent Information, 61, 101965, 2020.
• 20. Bekamiri, H., Hain, D.S., and Jurowetzki, R., PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT, https://arxiv.org/abs/2103.11933, Yayın tarihi Mar 22 2021, Güncelleme tarihi Mayıs 29 2021, Erişim tarihi Temmuz 25, 2023.
• 21. Chen, Y.L., and Chang, Y.C. A three-phase method for patent classification, Information Processing & Management (Inf. Process. Manage), 48 (6), 1017–1030, 2012.
• 22. Yücesoy, S., Dereli, T., and Durmuşoglu, A., Patent Classification via Textual Analysis Which Sections to be Included? 2018 International Conference on Artificial Intelligence and Data Processing, IDAP 2018, Malatya, Turkey, 1-8, 28-30 Eylül 2018.
• 23. Trappey, A., Development of a patent document classification and search platform using a back-propagation network, Expert Systems with Applications (Expert Syst. Appl.), 31, (4), 755-765, 2006.
• 24. Benzineb, K., and Guyot, J., Automated Patent Classification, Current challenges in patent information retrieval, The Information Retrieval Series 29, Springer-Verlag Berlin Heidelberg, 239-261, 2011.
• 25. Rousu J., Saunders J, Szedmak, C., Shawe S., Taylor, J., Bennett, K.P., and Parrado-Hernández, E., Kernel-Based Learning of Hierarchical Multilabel Classification Models, Journal of Machine Learning Research, 7, 1601–1626, 2006.
• 26. Wu, C., Ken, Y., Huang, T, Patent classification system using a new hybrid genetic algorithm support vector machine, Applied Soft Computing 10, 4, 1164-1177, 2010.
• 27. Hu, J., Li, S., Hu, J., and Yang, G., A hierarchical feature extraction model for multi-label mechanical patent classification, Sustainability, 10 (1), 1-22, 2018.
• 28. Alla G. Kravets, Nikita Lebedev, and Maxim Legenchenko, Patents images retrieval and convolutional neural network training dataset quality improvement, IV International Research Conference Information Technologies in Science, Management, Social Sphere and Medicine ITSMSSM 2017, 287–293, 2017.
• 29. Wang, Y., Du, J., Shao, Y., Li, A., and Xu, X., A Patent Text Classification Method Based on Phrase-Context Fusion Feature.", Proceedings of 2021 Chinese Intelligent Automation Conference, Deng Z., Springer, Singapore, 157-164, 2022.
• 30. Kantar O., Kilimci Z.H., Deep learning based hybrid gold index (XAU/USD) direction forecast model, Journal of the Faculty of Engineering and Architecture of Gazi University, 38 (2), 1117–1128, 2022.
• 31. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, https://arxiv.org/abs/1910.01108, Yayın tarihi Ekim 2019, Güncelleme Tarihi, Mart 2020, Erişim Tarihi Temmuz 25, 2023.
• 32. Devlin, J., Chang, M.-W., Lee, K., Google, K.T., and Language, A.I. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, https://arxiv.org/abs/1810.04805, Yayın tarihi Ekim 2018, Güncelleme Tarihi May 2019, Erişim Tarihi Temmuz 25, 2023.
• 33. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V., and Allen, P.G. (2019) RoBERTa: A Robustly Optimized BERT Pretraining Approach, https://arxiv.org/abs/1907.11692, Yayın tarihi Temmuz 2019, Güncelleme Tarihi Temmuz 2019, Erişim Tarihi Temmuz 25, 2023.
• 34. Li, S., Hu, J., Cui, Y., and Hu, J., DeepPatent: patent classification with convolutional neural networks and word embedding. Scientometrics, 117 (2), 721–744, 2018.
• 35. Lim, S., and Kwon, Y., IPC multi-label classification applying the characteristics of patent documents, Advances in Computer Science and Ubiquitous Computing. UCAWSN CUTE CSA 2016, Park, J., Lecture Notes in Electrical Engineering, Springer, Singapore, 421, 166–172, 2017.
• 36. Hu, J., Li, S., Hu, J., and Yang, G., A Hierarchical Feature Extraction Model for Multi-Label Mechanical Patent Classification, Sustainability, 10 (1), 219,2018.
• 37. Lee, J.S., and Hsiang, J., Patent classification by fine-tuning BERT language model, World Patent Information, 61, 101965, 2020.
• 38. Fall, C.J., Törcsvári, A., Benzineb, K., and Karetka, G., Automated categorization in the international patent classification, ACM SIGIR Forum, 37 (1), 10–25, 2003.
• 39. Qiu, X., Huang, X.-J., Liu, Z., and Zhou, J., Hierarchical Text Classification with Latent Concepts, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: short papers 598–602, Portland, Oregon, 19-24 Haziran 2011.
• 40. Seneviratne, D., Geva, S., Zuccon, G., Ferraro, G., and Meireles, M., Linking patents to knowledge sources: A context matching technique using automatic patent classification, Proceedings of the 23rd Australasian Document Computing Symposium, 1-4, 2018.
• 41. Aiolli, F., Cardin, R., Sebastiani, F., and Sperduti, A., Preferential Text Classification: Learning Algorithms and Evaluation Measures, Information retrieval, 12, 559–580, 2009.
• 42. Chen, Y.L., Chang, Y.C., A three-phase method for patent classification. Information Processing & Management (Inf. Process. Manage), 48 (6), 1017–1030, 2021.
• 43. Altuntas, F., Gök, M.Ş., Analysis of patent documents with utility mining: a case study of wind energy technology. Kybernetes, 50 (9), 2548–2582, 2020.
• 44. Perez-Molina, E. (2018) The role of patent citations as a footprint of technology. Journal of the Association for Information Science and Technology (J. Assoc. Inf. Sci. Technol), 69 (4), 610–618,2018.
• 45. Minin, V.A., Zatsman, I.M., Khavanskov, V.A., and Shubnikov, S.K., Methods of indicator-based assessment of knowledge transfer from science to technology, Automatic Documentation and Mathematical Linguistics (Autom. Doc. Math. Ling.), 51 (4), 180–190, 2017.
• 46. Risch, J., and Krestel, R., Learning patent speak: Investigating domain-specific word embeddings, 13th International Conference on Digital Information Management, ICDIM 2018, Berlin, Germany, 63–68, 24-26 Eylül 2018.
• 47. Sun, C., Qiu, X., Xu, Y., and Huang, X., How to Fine-Tune BERT for Text Classification? Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, 194–206, 18-20 Ekim 2019.
• 48. Roudsari A.H., Jafar Afshar Lee, Wookey Lee S., PatentNet: multi-label classification of patent documents using deep learning based language understanding, Scientometrics, 127, 207–231, 2022
• 49. Wang, J., Zhang, Z., Feng, L., Lin, K.Y., and Liu, P., Development of technology opportunity analysis based on technology landscape by extending technology elements with BERT and TRIZ. Technological Forecasting and Social Change (Technol. Forecasting Social Change), 191, 122481, 2023.
• 50. Qiu, Z., and Wang, Z., What is your next invention? - A framework of mining technological development rules and assisting in designing new technologies based on BERT as well as patent citations, Computers in Industry (Comput Ind), 145, 103829, 2023.
• 51. Choi, S., Lee, H., Park, E., and Choi, S., Deep learning for patent landscaping using transformer and graph embedding, Technological Forecasting and Social Change (Technol. Forecasting Social Change), 175, 121413, 2022.
• 52. Vaswani, A., Brain, G., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I, Attention Is All You Need, Advances in neural information processing systems, 31st Conference on Neural Information Processing Systems (NIPS2017), Long Beach, CA, USA.1-11, 4-9 Aralık 2017.
• 53. Golubev, A.A., and Loukachevitch, N. V., Use of Bert Neural Network Models for Sentiment Analysis in Russian. Automatic Documentation and Mathematical Linguistics, 55 (1), 17–25, 2021.
• 54. Syed, M.H., and Chung, S.-T., MenuNER: Domain-Adapted BERT Based NER Approach for a Domain with Limited Dataset and Its Application to Food Menu Domain, 2021.
• 55. Ozan, Ş., Özdil, U., Taşar, D.E., Arslan, B., and Polat, G., BERT Modeli’nin Sınıflandırma Doğruluğunun Sıfır-Atış Öğrenmesi ile Artırılması. Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi, 14 (2), 99–108, 2021. 56. Sevli, O., and Kemaloğlu, N. (2021) Classification of Tweets about Disasters as Real or Unreal by Google BERT Model Abstract, Veri Bilim Dergisi, 4 (1), 31-37, 2021.
• 57. Wu, Y., Schuster, M., Chen, Z., Le, Q. V, Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, J. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, https://arxiv.org/abs/1609.08144, Yayın tarihi Eylül 26 2016, Güncelleme Tarihi Ekim 8 2016, Erişim Tarihi Temmuz 25, 2023.
• 58. Jiang, S., Benge, C., and King, W.C., BERTVision --A Parameter-Efficient Approach for Question Answering, https://arxiv.org/abs/1609.08144, Yayın tarihi Şubat 24 2022, Erişim Tarihi Temmuz 25, 2023, 2022
• 59. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R., and Research, G. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, https://arxiv.org/abs/1909.11942, Yayın tarihi Eylül 26 2019, Güncelleme Tarihi Şubat 9 2020, Erişim Tarihi Temmuz 25, 2023.
• 60. Martin, L., Muller, B., Ortiz Suárez, P.J., Dupont, Y., Romary, L., Villemonte De La Clergerie, É., Seddah, D., and Sagot, B. CamemBERT: a Tasty French Language Model, https://arxiv.org/abs/1911.03894, Yayın tarihi Kasım 10 2019, Güncelleme Tarihi Mayıs 21 2020, Erişim Tarihi Temmuz 25, 2023.
• 61. Gao, Z., Feng, A., Song, X., and Wu, X., Target-dependent sentiment classification with BERT. IEEE Access, 7, 154290–154299,2019.
• 62. The World Intellectual Property Organization (WIPO) International Classifications - IT Support Area, https://www.wipo.int/classifications/ipc/en/ITsupport/Version20210101/transformations/stats, Yayın tarihi Ocak 1 2015, Güncelleme Tarihi Aralık 27 2022, Erişim Tarihi Mayıs 3, 2023.
• 63. Aroyehun, S.T., Angel, J., Majumder, N., Gelbukh, A., and Hussain, A. (2021) Leveraging label hierarchy using transfer and multi-task learning: A case study on patent classification, Neurocomputing, 464, 421–431,2021.
• 64. Smith, L.N., Cyclical learning rates for training neural networks. Proceedings - 2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017, Santa Rosa, CA, USA 464–472, 24-31 Mart, 2017.
• 65. Popel, M., and Bojar, O., Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguistics, 110 (1), 43–70, 2018.
• 66. Tan, K.L., Lee, C.P., Anbananthen, K.S.M., and Lim, K.M., RoBERTa-LSTM: A Hybrid Model for Sentiment Analysis with Transformer and Recurrent Neural Network. IEEE Access, 10, 21517–21525, 2022.
• 67. Huang, W.C., Wu, C.H., Luo, S.B., Chen, K.Y., Wang, H.M., and Toda, T., Speech recognition by simply fine-tuning BERT. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Toronto, ON, Canada, 7343–734706, 06-11 Haziran, 2021.
• 68. Bozkır, R., Cicioğlu, M., Toğay, C., and Çalhan, A., The Effect of Flow Times on Machine Learning Algorithms in Flow Based Classification of Network Traffic, European Journal of Science and Technology Special Issue, 36, 276–283, 2022.
• 69. Hao, Y., Dong, L., Wei, F., Xu, K., Hao, Y., Dong, L., Wei, F., and Xu, K., Visualizing and Understanding the Effectiveness of BERT, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 4143–4152 Hong Kong, China, 3–7 Kasım, 2019.
• 70. Abdelgawad, L., Kluegl, P., Genc, E., Falkner, S., and Hutter, F. Optimizing Neural Networks for Patent Classification, Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 688-703, Würzburg, Germany, 16–20 Eylül, 2019.