DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI

Kazım Fırıldak; Gaffari Çelik; Muhammed Fatih Talu

doi:10.54365/adyumbd.1390894

EN TR

DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI

Abstract

Ortagonal Düzgünleştirme (OD), derin ağların aşırı öğrenme (overfitting), gradyan patlamansı/kaybolmasını engellemek için kullanılmaktadır. Literatürde derin öğrenme için geliştirilen OD yöntemlerinin çoğunda ağ ağırlıklarını birim dik vektörler olarak öğrenme amaçlanmaktadır. Bu makalede ağ ağırlıklarını ikili olarak gruplayarak birim dik öğrenmeye zorlayan fonksiyon, maliyet fonksiyonuna eklenmektedir. Bu yöntem yapay sinir ağlarında ve konvülasyonel sinir ağlarında çeşitli veri kümelerinde (yapay veri ve gerçek veri) test edilmektedir. Ayrıca önerilen yöntem, literatürde öne çıkan Yumuşak Ortagonal (SO), Çift Yumuşak Ortagonal (DSO), Karşılıklı Tutarlılık (MC) ve Spektral Sınırlı İzometri Özellikli (SRIP) gibi yöntemler ile doğruluk, yürütülme zamanı, hata oranı metriklerinde karşılaştırılmaktadır. Karşılaştırma sonucunda doğruluk metriğinde farklı veri kümelerinin kullanan ağlarda %1-%5 arasında iyileşme sağlanmaktadır. Önerilen yöntem, Cifar10 veri kümesinde Resnet 110 ağında 92,96 dan %93,90’a ve Resnet 28-10 %95,84’den %96,78’a test başarısını yükseltmektedir.

Keywords

Ortagonal Düzgünleştirme, Sinir Ağları, Evrişimsel Sinir Ağları

References

Szegedy C, Wei Liu, Yangqing Jia, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. 2015 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2015, p. 1–9. https://doi.org/10.1109/CVPR.2015.7298594.
Krizhevsky A, Hinton GE. ImageNet Classification with Deep Convolutional Neural Networks. Adv Neural Inf Process Syst 2012;1907–1105:1–9.
Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial nets. Adv. Neural Inf. Process. Syst., 2014.
Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks 2015.
Çelik G, Talu MF. Resizing and cleaning of histopathological images using generative adversarial networks. Phys A Stat Mech Its Appl 2019:122652. https://doi.org/10.1016/j.physa.2019.122652.
Iizuka O, Kanavati F, Kato K, Rambeau M, Arihiro K, Tsuneki M. Deep Learning Models for Histopathological Classification of Gastric and Colonic Epithelial Tumours. Sci Rep 2020; 10:1504. https://doi.org/10.1038/s41598-020-58467-9.
Akram T, Lodhi HMJ, Naqvi SR, Naeem S, Alhaisoni M, Ali M, et al. A multilevel features selection framework for skin lesion classification. Human-Centric Comput Inf Sci 2020;10:12. https://doi.org/10.1186/s13673-020-00216-y.
Xu W, Keshmiri S, Wang G. Adversarially Approximated Autoencoder for Image Generation and Manipulation. IEEE Trans Multimed 2019;21:2387–96. https://doi.org/10.1109/TMM.2019.2898777.
Turkoglu M, Hanbay D, Sengur A. Multi-model LSTM-based convolutional neural networks for detection of apple diseases and pests. J Ambient Intell Humaniz Comput 2019. https://doi.org/10.1007/s12652-019-01591-w.
Grießhaber D, Vu NT, Maucher J. Low-resource text classification using domain-adversarial learning. Comput Speech Lang 2020;62:101056. https://doi.org/10.1016/j.csl.2019.101056.

Yin Y, Li H, Fu W. Faster-YOLO: An accurate and faster object detection method. Digit Signal Process 2020;102:102756. https://doi.org/10.1016/j.dsp.2020.102756.
Wu Y, Zhang Z, Wang G. Unsupervised Deep Feature Transfer for Low Resolution Image Classification. 2019 IEEE/CVF Int. Conf. Comput. Vis. Work., IEEE; 2019, p. 1065–9. https://doi.org/10.1109/ICCVW.2019.00136.
Salimans T, Kingma DP. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Adv. Neural Inf. Process. Syst., 2016.
Huang L, Liu X, Lang B, Yu AW, Wang Y, Li B. Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks. 32nd AAAI Conf Artif Intell AAAI 2018 2017.
Martin CH, Mahoney MW. Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning 2018.
Xu X, Wang G, Sullivan A, Zhang Z. Towards Learning Affine-Invariant Representations via Data-Efficient CNNs 2019.
Wang J, Zeng Z, Hou ZG. Advances in neural networks. vol. 149. 2015. https://doi.org/10.1016/j.neucom.2014.07.026.
Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Teh YW, Titterington M, editors. Proc. Thirteen. Int. Conf. Artif. Intell. Stat., vol. 9, Chia Laguna Resort, Sardinia, Italy: PMLR; 2010, p. 249–56.
He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. 2016 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2016, p. 770–8. https://doi.org/10.1109/CVPR.2016.90.
Veit A, Wilber M, Belongie S. Residual Networks Behave Like Ensembles of Relatively Shallow Networks 2016.
Mishkin D, Matas J. All you need is a good init 2015.
Cui L, Zhai H, Lin H. A Novel Orthogonal Extreme Learning Machine for Regression and Classification Problems. Symmetry (Basel) 2019;11:1284. https://doi.org/10.3390/sym11101284.
Shin J, Koo B, Kim Y, Paik J. Deep Binary Classification via Multi-Resolution Network and Stochastic Orthogonality for Subcompact Vehicle Recognition. Sensors 2020;20:2715. https://doi.org/10.3390/s20092715.
Jia K, Li S, Wen Y, Liu T, Tao D. Orthogonal Deep Neural Networks. IEEE Trans Pattern Anal Mach Intell 2020:1–1. https://doi.org/10.1109/TPAMI.2019.2948352.
Zhang Z, Ma W, Wu Y, Wang G. Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters. 2020 IEEE Winter Conf. Appl. Comput. Vis., IEEE; 2020, p. 1044–8. https://doi.org/10.1109/WACV45572.2020.9093466.
Bansal N, Chen X, Wang Z. Can we gain more from orthogonality regularizations in training deep CNNs? Adv Neural Inf Process Syst 2018;2018-Decem:4261–71.
Xie D, Xiong J, Pu S. All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation. 2017 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2017, p. 5075–84. https://doi.org/10.1109/CVPR.2017.539.
Zhang L, Li D, Guo Q. Deep Learning From Spatio-Temporal Data Using Orthogonal Regularizaion Residual CNN for Air Prediction. IEEE Access 2020;8:66037–47. https://doi.org/10.1109/ACCESS.2020.2985657.
Zhu F, Liang Q. OCRNN: An orthogonal constrained recurrent neural network for sleep analysis based on EEG data. Ad Hoc Networks 2020;104:102178. https://doi.org/10.1016/j.adhoc.2020.102178.
Zhang J, Yao R, Ge W, Gao J. Orthogonal convolutional neural networks for automatic sleep stage classification based on single-channel EEG. Comput Methods Programs Biomed 2020;183:105089. https://doi.org/10.1016/j.cmpb.2019.105089.
Jalwana MAAK, Akhtar N, Bennamoun M, Mian A. Orthogonal Deep Models as Defense Against Black-Box Attacks. IEEE Access 2020;8:119744–57. https://doi.org/10.1109/ACCESS.2020.3005961.
Rodríguez P, Gonzàlez J, Cucurull G, Gonfaus JM, Roca X. Regularizing CNNs with Locally Constrained Decorrelations 2016.
Balestriero R, richard baraniuk. A Spline Theory of Deep Learning. In: Dy J, Krause A, editors. Proc. 35th Int. Conf. Mach. Learn., vol. 80, PMLR; 2018, p. 374–83.
Donoho DL. Compressed sensing. IEEE Trans Inf Theory 2006;52:1289–306. https://doi.org/10.1109/TIT.2006.871582.
Lu C, Li H, Lin Z. Optimized Projections for Compressed Sensing via Direct Mutual Coherence Minimization 2015.
Yoshida Y, Miyato T. Spectral Norm Regularization for Improving the Generalizability of Deep Learning 2017.
Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition 2014.
Griewank A, Walther A. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM; 2008.
Tuncer İH, Kaya H, Tiftikci H. OTOMATİK TÜREV ARAÇLARI İLE AYRIK ADJOINT ÇÖZÜCÜ GELİŞTİRİLMESİ. 7. Havacılık ve Uzay Konf., Samsun/Türkiye: 2018, p. 1.
D. M. W. Powers. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness & correlation 2011. https://doi.org/10.9735/2229-3981.
Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86:2278–324. https://doi.org/10.1109/5.726791.
Xiao H, Rasul K, Vollgraf R. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms 2017.
Zagoruyko S, Komodakis N. Wide Residual Networks 2016.
Kingma DP, Ba J. Adam: A Method for Stochastic Optimization 2014.

Details

Primary Language

Turkish

Subjects

Pattern Recognition

Journal Section

Research Article

Authors

Kazım Fırıldak ^*
0000-0002-1958-3627
Türkiye

Gaffari Çelik
0000-0001-5658-9529
Türkiye

Muhammed Fatih Talu
0000-0003-1166-8404
Türkiye

Publication Date

April 30, 2024

Submission Date

November 14, 2023

Acceptance Date

April 22, 2024

Published in Issue

Year 2024 Volume: 11 Number: 22

DOI

https://doi.org/10.54365/adyumbd.1390894

IZ

https://izlik.org/JA85DF48UU

APA

Fırıldak, K., Çelik, G., & Talu, M. F. (2024). DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi, 11(22), 18-34. https://doi.org/10.54365/adyumbd.1390894

AMA

1.Fırıldak K, Çelik G, Talu MF. DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi. 2024;11(22):18-34. doi:10.54365/adyumbd.1390894

Chicago

Fırıldak, Kazım, Gaffari Çelik, and Muhammed Fatih Talu. 2024. “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi 11 (22): 18-34. https://doi.org/10.54365/adyumbd.1390894.

EndNote

Fırıldak K, Çelik G, Talu MF (April 1, 2024) DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi 11 22 18–34.

IEEE

[1]K. Fırıldak, G. Çelik, and M. F. Talu, “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”, Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi, vol. 11, no. 22, pp. 18–34, Apr. 2024, doi: 10.54365/adyumbd.1390894.

ISNAD

Fırıldak, Kazım - Çelik, Gaffari - Talu, Muhammed Fatih. “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi 11/22 (April 1, 2024): 18-34. https://doi.org/10.54365/adyumbd.1390894.

JAMA

1.Fırıldak K, Çelik G, Talu MF. DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi. 2024;11:18–34.

MLA

Fırıldak, Kazım, et al. “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi, vol. 11, no. 22, Apr. 2024, pp. 18-34, doi:10.54365/adyumbd.1390894.

Vancouver

1.Kazım Fırıldak, Gaffari Çelik, Muhammed Fatih Talu. DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi. 2024 Apr. 1;11(22):18-34. doi:10.54365/adyumbd.1390894

Cited By

A Residual Neural Network with a Novel Orthogonal Regularization for Covid-19 Diagnosis using X-ray images

Türk Doğa ve Fen Dergisi

https://doi.org/10.46810/tdfd.1661900

A NEW ORTOGONAL REGULARIZATION APPROACH FOR DEEP NETWORKS

Abstract

Keywords

DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI

Abstract

Keywords

References

Details

Primary Language

Subjects

Journal Section

Authors

Publication Date

Submission Date

Acceptance Date

Published in Issue

DOI

IZ

Cite

Cited By

A Residual Neural Network with a Novel Orthogonal Regularization for Covid-19 Diagnosis using X-ray images