Ön eğitimli Bert modeli ile patent sınıflandırılması

Selen Yücesoy Kahraman; Alptekin Durmuşoğlu; Türkay Dereli

doi:10.17341/gazimmfd.1292543

Ön eğitimli Bert modeli ile patent sınıflandırılması

Öz

Patentler, bilgi teknolojilerindeki yeniliklerin korunmasına yardımcı olan ve bu yeniliklerin yaratıcısına belirli bir süre boyunca özel haklar sağlayan belgelerdir. Bu haklar, patent sahibine yeniliği ticari olarak kullanma hakkı verirken, başkalarının yeniliği izinsiz kullanmasını engeller. Radikal yenilikler ve çığır açan teknolojik gelişmeler, mevcut patentlerde yer alan teknik bilgilerden türetilmiştir. Otomatik bir sınıflandırma sistemi kullanılarak, ait oldukları teknik sınıfa atanan patentler, araştırmacıların önünü açabilmekte ve yeni buluşlar yaratabilecekleri bir ortam sağlayabilmektedir. Bu çalışma, BERT algoritmasını kullanarak otomatik bir patent sınıflandırma analizi sunmaktadır. Otomatik patent sınıflandırma problemlerinde daha başarılı tahmin doğruluğuna ulaşabilmek için yapılan hiper parametre analizleri bu çalışmada da tercih edilmiştir. Elde edilen sonuçlar literatürdeki sonuçlarla rekabet edecek düzeydedir. Bu çalışmada alt sınıf düzeyinde % 55,9 tahmin doğruluğu elde edilmiştir.

Anahtar Kelimeler

Destekleyen Kurum

Destekleyen bir kurum bulunmamaktadır.

Proje Numarası

Proje numarası bulunmamaktadır.

Kaynakça

1. Asche, G., “80% of technical information found only in patents” – Is there proof of this ?, World Patent Information, 48, 16–28, 2017.
2. Seo, W. A patent-based approach to identifying potential technology opportunities realizable from a firm’s internal capabilities, Computers & Industrial Engineering (Comput. Ind. Eng), 171, 108395, 2022.
3. Sofean, M., Deep learning based pipeline with multichannel inputs for patent classification, World Patent Information, 66, 102060, 2021.
4. Giordano V., Chiarello F., Melluso N., Fantoni G. and Bonaccorsi A., Text and Dynamic Network Analysis for Measuring Technological Convergence: A Case Study on Defense Patent Data, IEEE Transactions on Engineering Management (IEEE Trans. Eng. Manage), 70, 4, 1490-1503, 2023.
5. Durmusoglu, A., Updating technology forecasting models using statistical control charts, Kybernetes, 47 (4), 672–688, 2018.
6. Hain, D.S., Jurowetzki, R., Buchmann, T., and Wolf, P. A., Text-Embedding-based Approach to Measure Patent-to-Patent Technological Similarity, Technological Forecasting and Social Change (Technol. Forecasting Social Change), 177, 121559, 2022.
7. Yan, B., and Luo, J., Measuring technological distance for patent mapping. The Journal of the Association for Information Science and Technology, 68 (2), 423–437, 2017.
8. Erzurumlu, S.S., and Pachamanova, D., Topic modeling and technology forecasting for assessing the commercial viability of healthcare innovations. Technological Forecasting and Social Change (Technol. Forecasting Social Change), 156, 120041, 2020.

9. Grawe, M.F., Martins, C.A., and Bonfante, A.G., Automated patent classification using word embedding, Proceedings - 16th IEEE International Conference on Machine Learning and Applications, ICMLA 2017, Cancun, Mexico, 408–411, 18-21 Aralık 2017.
10. Altuntas, S., Dereli, T., and Kusiak, A., Analysis of patent documents with weighted association rules, Technological Forecasting and Social Change (Technol. Forecasting Social Change), 92, 249–262, 2015.
11. Mutlu N.G., Altuntaş S., Monitoring technological changes with statistical control charts based on patent data, Journal of the Faculty of Engineering and Architecture of Gazi University, 36 (4), 1875-1892, 2021.
12. Li, S., Hu, J., Cui, Y., and Hu, J., DeepPatent: patent classification with convolutional neural networks and word embedding, Scientometrics, 117, 721–744, 2018.
13. Yücesoy Kahraman, S., Dereli, T., Durmuşoglu, A. Forty years of automated patent classification, International Journal of Information Technology & Decision Making (Int. J. Inf. Technol. Decis. Making), 2023. https://doi.org/10.1142/S0219622023500165
14. Fall, C.J., Törcsvári, A., Fiévet, P., and Karetka, G., Automated categorization of German-language patent documents, Expert Systems with Applications (Expert Syst. Appl), 26 (2), 269–277, 2004.
15. Fall, C., and Benzineb, K., Literature survey: Issues to be considered in the automatic classification of patents, World Intellectual Property Organization, 2002.
16. Piroi, F., and Hanbury, A., Multilingual Patent Text Retrieval Evaluation: CLEF–IP, Information Retrieval Evaluation in a Changing World, The Information Retrieval Series, Springer Nature Switzerland AG, 41, 365–387, 2019.
17. Roda, G., Tait, J., Piroi, F., and Zenz, V., CLEF-IP 2009: Retrieval Experiments in the Intellectual Property Domain, Multilingual Information Access Evaluation I. Text Retrieval Experiments CLEF 2009, Lecture Notes in Computer Science, 6241. Springer, Berlin, Heidelberg, 2010.
18. Karaca A., Aydın Ö., Generating headlines for Turkish news texts with transformer architecture based deep learning method, Journal of the Faculty of Engineering and Architecture of Gazi University, 39 (1), 485–496, 2023.
19. Lee, J.S., and Hsiang, J., Patent classification by fine-tuning BERT language model, World Patent Information, 61, 101965, 2020.
20. Bekamiri, H., Hain, D.S., and Jurowetzki, R., PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT, https://arxiv.org/abs/2103.11933, Yayın tarihi Mar 22 2021, Güncelleme tarihi Mayıs 29 2021, Erişim tarihi Temmuz 25, 2023.
21. Chen, Y.L., and Chang, Y.C. A three-phase method for patent classification, Information Processing & Management (Inf. Process. Manage), 48 (6), 1017–1030, 2012.
22. Yücesoy, S., Dereli, T., and Durmuşoglu, A., Patent Classification via Textual Analysis Which Sections to be Included? 2018 International Conference on Artificial Intelligence and Data Processing, IDAP 2018, Malatya, Turkey, 1-8, 28-30 Eylül 2018.
23. Trappey, A., Development of a patent document classification and search platform using a back-propagation network, Expert Systems with Applications (Expert Syst. Appl.), 31, (4), 755-765, 2006.
24. Benzineb, K., and Guyot, J., Automated Patent Classification, Current challenges in patent information retrieval, The Information Retrieval Series 29, Springer-Verlag Berlin Heidelberg, 239-261, 2011.
25. Rousu J., Saunders J, Szedmak, C., Shawe S., Taylor, J., Bennett, K.P., and Parrado-Hernández, E., Kernel-Based Learning of Hierarchical Multilabel Classification Models, Journal of Machine Learning Research, 7, 1601–1626, 2006.
26. Wu, C., Ken, Y., Huang, T, Patent classification system using a new hybrid genetic algorithm support vector machine, Applied Soft Computing 10, 4, 1164-1177, 2010.
27. Hu, J., Li, S., Hu, J., and Yang, G., A hierarchical feature extraction model for multi-label mechanical patent classification, Sustainability, 10 (1), 1-22, 2018.
28. Alla G. Kravets, Nikita Lebedev, and Maxim Legenchenko, Patents images retrieval and convolutional neural network training dataset quality improvement, IV International Research Conference Information Technologies in Science, Management, Social Sphere and Medicine ITSMSSM 2017, 287–293, 2017.
29. Wang, Y., Du, J., Shao, Y., Li, A., and Xu, X., A Patent Text Classification Method Based on Phrase-Context Fusion Feature.", Proceedings of 2021 Chinese Intelligent Automation Conference, Deng Z., Springer, Singapore, 157-164, 2022.
30. Kantar O., Kilimci Z.H., Deep learning based hybrid gold index (XAU/USD) direction forecast model, Journal of the Faculty of Engineering and Architecture of Gazi University, 38 (2), 1117–1128, 2022.
31. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, https://arxiv.org/abs/1910.01108, Yayın tarihi Ekim 2019, Güncelleme Tarihi, Mart 2020, Erişim Tarihi Temmuz 25, 2023.
32. Devlin, J., Chang, M.-W., Lee, K., Google, K.T., and Language, A.I. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, https://arxiv.org/abs/1810.04805, Yayın tarihi Ekim 2018, Güncelleme Tarihi May 2019, Erişim Tarihi Temmuz 25, 2023.
33. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V., and Allen, P.G. (2019) RoBERTa: A Robustly Optimized BERT Pretraining Approach, https://arxiv.org/abs/1907.11692, Yayın tarihi Temmuz 2019, Güncelleme Tarihi Temmuz 2019, Erişim Tarihi Temmuz 25, 2023.
34. Li, S., Hu, J., Cui, Y., and Hu, J., DeepPatent: patent classification with convolutional neural networks and word embedding. Scientometrics, 117 (2), 721–744, 2018.
35. Lim, S., and Kwon, Y., IPC multi-label classification applying the characteristics of patent documents, Advances in Computer Science and Ubiquitous Computing. UCAWSN CUTE CSA 2016, Park, J., Lecture Notes in Electrical Engineering, Springer, Singapore, 421, 166–172, 2017.
36. Hu, J., Li, S., Hu, J., and Yang, G., A Hierarchical Feature Extraction Model for Multi-Label Mechanical Patent Classification, Sustainability, 10 (1), 219,2018.
37. Lee, J.S., and Hsiang, J., Patent classification by fine-tuning BERT language model, World Patent Information, 61, 101965, 2020.
38. Fall, C.J., Törcsvári, A., Benzineb, K., and Karetka, G., Automated categorization in the international patent classification, ACM SIGIR Forum, 37 (1), 10–25, 2003.
39. Qiu, X., Huang, X.-J., Liu, Z., and Zhou, J., Hierarchical Text Classification with Latent Concepts, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: short papers 598–602, Portland, Oregon, 19-24 Haziran 2011.
40. Seneviratne, D., Geva, S., Zuccon, G., Ferraro, G., and Meireles, M., Linking patents to knowledge sources: A context matching technique using automatic patent classification, Proceedings of the 23rd Australasian Document Computing Symposium, 1-4, 2018.
41. Aiolli, F., Cardin, R., Sebastiani, F., and Sperduti, A., Preferential Text Classification: Learning Algorithms and Evaluation Measures, Information retrieval, 12, 559–580, 2009.
42. Chen, Y.L., Chang, Y.C., A three-phase method for patent classification. Information Processing & Management (Inf. Process. Manage), 48 (6), 1017–1030, 2021.
43. Altuntas, F., Gök, M.Ş., Analysis of patent documents with utility mining: a case study of wind energy technology. Kybernetes, 50 (9), 2548–2582, 2020.
44. Perez-Molina, E. (2018) The role of patent citations as a footprint of technology. Journal of the Association for Information Science and Technology (J. Assoc. Inf. Sci. Technol), 69 (4), 610–618,2018.
45. Minin, V.A., Zatsman, I.M., Khavanskov, V.A., and Shubnikov, S.K., Methods of indicator-based assessment of knowledge transfer from science to technology, Automatic Documentation and Mathematical Linguistics (Autom. Doc. Math. Ling.), 51 (4), 180–190, 2017.
46. Risch, J., and Krestel, R., Learning patent speak: Investigating domain-specific word embeddings, 13th International Conference on Digital Information Management, ICDIM 2018, Berlin, Germany, 63–68, 24-26 Eylül 2018.
47. Sun, C., Qiu, X., Xu, Y., and Huang, X., How to Fine-Tune BERT for Text Classification? Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, 194–206, 18-20 Ekim 2019.
48. Roudsari A.H., Jafar Afshar Lee, Wookey Lee S., PatentNet: multi-label classification of patent documents using deep learning based language understanding, Scientometrics, 127, 207–231, 2022
49. Wang, J., Zhang, Z., Feng, L., Lin, K.Y., and Liu, P., Development of technology opportunity analysis based on technology landscape by extending technology elements with BERT and TRIZ. Technological Forecasting and Social Change (Technol. Forecasting Social Change), 191, 122481, 2023.
50. Qiu, Z., and Wang, Z., What is your next invention? - A framework of mining technological development rules and assisting in designing new technologies based on BERT as well as patent citations, Computers in Industry (Comput Ind), 145, 103829, 2023.
51. Choi, S., Lee, H., Park, E., and Choi, S., Deep learning for patent landscaping using transformer and graph embedding, Technological Forecasting and Social Change (Technol. Forecasting Social Change), 175, 121413, 2022.
52. Vaswani, A., Brain, G., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I, Attention Is All You Need, Advances in neural information processing systems, 31st Conference on Neural Information Processing Systems (NIPS2017), Long Beach, CA, USA.1-11, 4-9 Aralık 2017.
53. Golubev, A.A., and Loukachevitch, N. V., Use of Bert Neural Network Models for Sentiment Analysis in Russian. Automatic Documentation and Mathematical Linguistics, 55 (1), 17–25, 2021.
54. Syed, M.H., and Chung, S.-T., MenuNER: Domain-Adapted BERT Based NER Approach for a Domain with Limited Dataset and Its Application to Food Menu Domain, 2021.
55. Ozan, Ş., Özdil, U., Taşar, D.E., Arslan, B., and Polat, G., BERT Modeli’nin Sınıflandırma Doğruluğunun Sıfır-Atış Öğrenmesi ile Artırılması. Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi, 14 (2), 99–108, 2021. 56. Sevli, O., and Kemaloğlu, N. (2021) Classification of Tweets about Disasters as Real or Unreal by Google BERT Model Abstract, Veri Bilim Dergisi, 4 (1), 31-37, 2021.
57. Wu, Y., Schuster, M., Chen, Z., Le, Q. V, Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, J. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, https://arxiv.org/abs/1609.08144, Yayın tarihi Eylül 26 2016, Güncelleme Tarihi Ekim 8 2016, Erişim Tarihi Temmuz 25, 2023.
58. Jiang, S., Benge, C., and King, W.C., BERTVision --A Parameter-Efficient Approach for Question Answering, https://arxiv.org/abs/1609.08144, Yayın tarihi Şubat 24 2022, Erişim Tarihi Temmuz 25, 2023, 2022
59. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R., and Research, G. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, https://arxiv.org/abs/1909.11942, Yayın tarihi Eylül 26 2019, Güncelleme Tarihi Şubat 9 2020, Erişim Tarihi Temmuz 25, 2023.
60. Martin, L., Muller, B., Ortiz Suárez, P.J., Dupont, Y., Romary, L., Villemonte De La Clergerie, É., Seddah, D., and Sagot, B. CamemBERT: a Tasty French Language Model, https://arxiv.org/abs/1911.03894, Yayın tarihi Kasım 10 2019, Güncelleme Tarihi Mayıs 21 2020, Erişim Tarihi Temmuz 25, 2023.
61. Gao, Z., Feng, A., Song, X., and Wu, X., Target-dependent sentiment classification with BERT. IEEE Access, 7, 154290–154299,2019.
62. The World Intellectual Property Organization (WIPO) International Classifications - IT Support Area, https://www.wipo.int/classifications/ipc/en/ITsupport/Version20210101/transformations/stats, Yayın tarihi Ocak 1 2015, Güncelleme Tarihi Aralık 27 2022, Erişim Tarihi Mayıs 3, 2023.
63. Aroyehun, S.T., Angel, J., Majumder, N., Gelbukh, A., and Hussain, A. (2021) Leveraging label hierarchy using transfer and multi-task learning: A case study on patent classification, Neurocomputing, 464, 421–431,2021.
64. Smith, L.N., Cyclical learning rates for training neural networks. Proceedings - 2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017, Santa Rosa, CA, USA 464–472, 24-31 Mart, 2017.
65. Popel, M., and Bojar, O., Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguistics, 110 (1), 43–70, 2018.
66. Tan, K.L., Lee, C.P., Anbananthen, K.S.M., and Lim, K.M., RoBERTa-LSTM: A Hybrid Model for Sentiment Analysis with Transformer and Recurrent Neural Network. IEEE Access, 10, 21517–21525, 2022.
67. Huang, W.C., Wu, C.H., Luo, S.B., Chen, K.Y., Wang, H.M., and Toda, T., Speech recognition by simply fine-tuning BERT. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Toronto, ON, Canada, 7343–734706, 06-11 Haziran, 2021.
68. Bozkır, R., Cicioğlu, M., Toğay, C., and Çalhan, A., The Effect of Flow Times on Machine Learning Algorithms in Flow Based Classification of Network Traffic, European Journal of Science and Technology Special Issue, 36, 276–283, 2022.
69. Hao, Y., Dong, L., Wei, F., Xu, K., Hao, Y., Dong, L., Wei, F., and Xu, K., Visualizing and Understanding the Effectiveness of BERT, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 4143–4152 Hong Kong, China, 3–7 Kasım, 2019.
70. Abdelgawad, L., Kluegl, P., Genc, E., Falkner, S., and Hutter, F. Optimizing Neural Networks for Patent Classification, Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 688-703, Würzburg, Germany, 16–20 Eylül, 2019.

Ayrıntılar

Birincil Dil

Türkçe

Konular

Mühendislik

Bölüm

Araştırma Makalesi

Yazarlar

Selen Yücesoy Kahraman ^*
0000-0002-0284-5133
Türkiye

Alptekin Durmuşoğlu
0000-0001-9800-5747
Türkiye

Türkay Dereli
0000-0002-2130-5503
Türkiye

Erken Görünüm Tarihi

17 Mayıs 2024

Yayımlanma Tarihi

20 Mayıs 2024

Gönderilme Tarihi

4 Mayıs 2023

Kabul Tarihi

21 Ekim 2023

Yayımlandığı Sayı

Yıl 2024 Cilt: 39 Sayı: 4

DOI

https://doi.org/10.17341/gazimmfd.1292543

IZ

https://izlik.org/JA26EE32EZ

Kaynak Göster

RIS / Bibtex

APA

Yücesoy Kahraman, S., Durmuşoğlu, A., & Dereli, T. (2024). Ön eğitimli Bert modeli ile patent sınıflandırılması. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 39(4), 2484-2496. https://doi.org/10.17341/gazimmfd.1292543

AMA

1.Yücesoy Kahraman S, Durmuşoğlu A, Dereli T. Ön eğitimli Bert modeli ile patent sınıflandırılması. GUMMFD. 2024;39(4):2484-2496. doi:10.17341/gazimmfd.1292543

Chicago

Yücesoy Kahraman, Selen, Alptekin Durmuşoğlu, ve Türkay Dereli. 2024. “Ön eğitimli Bert modeli ile patent sınıflandırılması”. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi 39 (4): 2484-96. https://doi.org/10.17341/gazimmfd.1292543.

EndNote

Yücesoy Kahraman S, Durmuşoğlu A, Dereli T (01 Mayıs 2024) Ön eğitimli Bert modeli ile patent sınıflandırılması. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi 39 4 2484–2496.

IEEE

[1]S. Yücesoy Kahraman, A. Durmuşoğlu, ve T. Dereli, “Ön eğitimli Bert modeli ile patent sınıflandırılması”, GUMMFD, c. 39, sy 4, ss. 2484–2496, May. 2024, doi: 10.17341/gazimmfd.1292543.

ISNAD

Yücesoy Kahraman, Selen - Durmuşoğlu, Alptekin - Dereli, Türkay. “Ön eğitimli Bert modeli ile patent sınıflandırılması”. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi 39/4 (01 Mayıs 2024): 2484-2496. https://doi.org/10.17341/gazimmfd.1292543.

JAMA

1.Yücesoy Kahraman S, Durmuşoğlu A, Dereli T. Ön eğitimli Bert modeli ile patent sınıflandırılması. GUMMFD. 2024;39:2484–2496.

MLA

Yücesoy Kahraman, Selen, vd. “Ön eğitimli Bert modeli ile patent sınıflandırılması”. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, c. 39, sy 4, Mayıs 2024, ss. 2484-96, doi:10.17341/gazimmfd.1292543.

Vancouver

1.Selen Yücesoy Kahraman, Alptekin Durmuşoğlu, Türkay Dereli. Ön eğitimli Bert modeli ile patent sınıflandırılması. GUMMFD. 01 Mayıs 2024;39(4):2484-96. doi:10.17341/gazimmfd.1292543

Ön eğitimli Bert modeli ile patent sınıflandırılması

Ön eğitimli Bert modeli ile patent sınıflandırılması

Öz

Anahtar Kelimeler

Destekleyen Kurum

Proje Numarası

Kaynakça

Ayrıntılar

Birincil Dil

Konular

Bölüm

Yazarlar

Erken Görünüm Tarihi

Yayımlanma Tarihi

Gönderilme Tarihi

Kabul Tarihi

Yayımlandığı Sayı

DOI

IZ

Kaynak Göster

Cited By

Classification and Analysis of Employee Feedback with Deep Learning Algorithms

Dijital mimarlık bilgisinin tasnif ve temsili: Bir çevrim içi ansiklopedi modeli önerisi

Integrating Generative Artificial Intelligence techniques into technology function matrix analysis

Türkçe soru cevaplama için büyük dil modelleri üzerinde geniş ölçekli etki analizi

Multi-Class News Classification with BERT, DistilBERT, RoBERTa, and ELECTRA Natural Language Processing Models