Research Article
BibTex RIS Cite

DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI

Year 2024, , 18 - 34, 30.04.2024

Kazım Fırıldak , Gaffari Çelik , Muhammed Fatih Talu

https://doi.org/10.54365/adyumbd.1390894

Abstract

Ortagonal Düzgünleştirme (OD), derin ağların aşırı öğrenme (overfitting), gradyan patlamansı/kaybolmasını engellemek için kullanılmaktadır. Literatürde derin öğrenme için geliştirilen OD yöntemlerinin çoğunda ağ ağırlıklarını birim dik vektörler olarak öğrenme amaçlanmaktadır. Bu makalede ağ ağırlıklarını ikili olarak gruplayarak birim dik öğrenmeye zorlayan fonksiyon, maliyet fonksiyonuna eklenmektedir. Bu yöntem yapay sinir ağlarında ve konvülasyonel sinir ağlarında çeşitli veri kümelerinde (yapay veri ve gerçek veri) test edilmektedir. Ayrıca önerilen yöntem, literatürde öne çıkan Yumuşak Ortagonal (SO), Çift Yumuşak Ortagonal (DSO), Karşılıklı Tutarlılık (MC) ve Spektral Sınırlı İzometri Özellikli (SRIP) gibi yöntemler ile doğruluk, yürütülme zamanı, hata oranı metriklerinde karşılaştırılmaktadır. Karşılaştırma sonucunda doğruluk metriğinde farklı veri kümelerinin kullanan ağlarda %1-%5 arasında iyileşme sağlanmaktadır. Önerilen yöntem, Cifar10 veri kümesinde Resnet 110 ağında 92,96 dan %93,90’a ve Resnet 28-10 %95,84’den %96,78’a test başarısını yükseltmektedir.

Keywords

Ortagonal Düzgünleştirme Sinir Ağları Evrişimsel Sinir Ağları

References

  • Szegedy C, Wei Liu, Yangqing Jia, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. 2015 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2015, p. 1–9. https://doi.org/10.1109/CVPR.2015.7298594.
  • Krizhevsky A, Hinton GE. ImageNet Classification with Deep Convolutional Neural Networks. Adv Neural Inf Process Syst 2012;1907–1105:1–9.
  • Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial nets. Adv. Neural Inf. Process. Syst., 2014.
  • Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks 2015.
  • Çelik G, Talu MF. Resizing and cleaning of histopathological images using generative adversarial networks. Phys A Stat Mech Its Appl 2019:122652. https://doi.org/10.1016/j.physa.2019.122652.
  • Iizuka O, Kanavati F, Kato K, Rambeau M, Arihiro K, Tsuneki M. Deep Learning Models for Histopathological Classification of Gastric and Colonic Epithelial Tumours. Sci Rep 2020; 10:1504. https://doi.org/10.1038/s41598-020-58467-9.
  • Akram T, Lodhi HMJ, Naqvi SR, Naeem S, Alhaisoni M, Ali M, et al. A multilevel features selection framework for skin lesion classification. Human-Centric Comput Inf Sci 2020;10:12. https://doi.org/10.1186/s13673-020-00216-y.
  • Xu W, Keshmiri S, Wang G. Adversarially Approximated Autoencoder for Image Generation and Manipulation. IEEE Trans Multimed 2019;21:2387–96. https://doi.org/10.1109/TMM.2019.2898777.
  • Turkoglu M, Hanbay D, Sengur A. Multi-model LSTM-based convolutional neural networks for detection of apple diseases and pests. J Ambient Intell Humaniz Comput 2019. https://doi.org/10.1007/s12652-019-01591-w.
  • Grießhaber D, Vu NT, Maucher J. Low-resource text classification using domain-adversarial learning. Comput Speech Lang 2020;62:101056. https://doi.org/10.1016/j.csl.2019.101056.
  • Yin Y, Li H, Fu W. Faster-YOLO: An accurate and faster object detection method. Digit Signal Process 2020;102:102756. https://doi.org/10.1016/j.dsp.2020.102756.
  • Wu Y, Zhang Z, Wang G. Unsupervised Deep Feature Transfer for Low Resolution Image Classification. 2019 IEEE/CVF Int. Conf. Comput. Vis. Work., IEEE; 2019, p. 1065–9. https://doi.org/10.1109/ICCVW.2019.00136.
  • Salimans T, Kingma DP. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Adv. Neural Inf. Process. Syst., 2016.
  • Huang L, Liu X, Lang B, Yu AW, Wang Y, Li B. Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks. 32nd AAAI Conf Artif Intell AAAI 2018 2017.
  • Martin CH, Mahoney MW. Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning 2018.
  • Xu X, Wang G, Sullivan A, Zhang Z. Towards Learning Affine-Invariant Representations via Data-Efficient CNNs 2019.
  • Wang J, Zeng Z, Hou ZG. Advances in neural networks. vol. 149. 2015. https://doi.org/10.1016/j.neucom.2014.07.026.
  • Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Teh YW, Titterington M, editors. Proc. Thirteen. Int. Conf. Artif. Intell. Stat., vol. 9, Chia Laguna Resort, Sardinia, Italy: PMLR; 2010, p. 249–56.
  • He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. 2016 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2016, p. 770–8. https://doi.org/10.1109/CVPR.2016.90.
  • Veit A, Wilber M, Belongie S. Residual Networks Behave Like Ensembles of Relatively Shallow Networks 2016.
  • Mishkin D, Matas J. All you need is a good init 2015.
  • Cui L, Zhai H, Lin H. A Novel Orthogonal Extreme Learning Machine for Regression and Classification Problems. Symmetry (Basel) 2019;11:1284. https://doi.org/10.3390/sym11101284.
  • Shin J, Koo B, Kim Y, Paik J. Deep Binary Classification via Multi-Resolution Network and Stochastic Orthogonality for Subcompact Vehicle Recognition. Sensors 2020;20:2715. https://doi.org/10.3390/s20092715.
  • Jia K, Li S, Wen Y, Liu T, Tao D. Orthogonal Deep Neural Networks. IEEE Trans Pattern Anal Mach Intell 2020:1–1. https://doi.org/10.1109/TPAMI.2019.2948352.
  • Zhang Z, Ma W, Wu Y, Wang G. Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters. 2020 IEEE Winter Conf. Appl. Comput. Vis., IEEE; 2020, p. 1044–8. https://doi.org/10.1109/WACV45572.2020.9093466.
  • Bansal N, Chen X, Wang Z. Can we gain more from orthogonality regularizations in training deep CNNs? Adv Neural Inf Process Syst 2018;2018-Decem:4261–71.
  • Xie D, Xiong J, Pu S. All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation. 2017 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2017, p. 5075–84. https://doi.org/10.1109/CVPR.2017.539.
  • Zhang L, Li D, Guo Q. Deep Learning From Spatio-Temporal Data Using Orthogonal Regularizaion Residual CNN for Air Prediction. IEEE Access 2020;8:66037–47. https://doi.org/10.1109/ACCESS.2020.2985657.
  • Zhu F, Liang Q. OCRNN: An orthogonal constrained recurrent neural network for sleep analysis based on EEG data. Ad Hoc Networks 2020;104:102178. https://doi.org/10.1016/j.adhoc.2020.102178.
  • Zhang J, Yao R, Ge W, Gao J. Orthogonal convolutional neural networks for automatic sleep stage classification based on single-channel EEG. Comput Methods Programs Biomed 2020;183:105089. https://doi.org/10.1016/j.cmpb.2019.105089.
  • Jalwana MAAK, Akhtar N, Bennamoun M, Mian A. Orthogonal Deep Models as Defense Against Black-Box Attacks. IEEE Access 2020;8:119744–57. https://doi.org/10.1109/ACCESS.2020.3005961.
  • Rodríguez P, Gonzàlez J, Cucurull G, Gonfaus JM, Roca X. Regularizing CNNs with Locally Constrained Decorrelations 2016.
  • Balestriero R, richard baraniuk. A Spline Theory of Deep Learning. In: Dy J, Krause A, editors. Proc. 35th Int. Conf. Mach. Learn., vol. 80, PMLR; 2018, p. 374–83.
  • Donoho DL. Compressed sensing. IEEE Trans Inf Theory 2006;52:1289–306. https://doi.org/10.1109/TIT.2006.871582.
  • Lu C, Li H, Lin Z. Optimized Projections for Compressed Sensing via Direct Mutual Coherence Minimization 2015.
  • Yoshida Y, Miyato T. Spectral Norm Regularization for Improving the Generalizability of Deep Learning 2017.
  • Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition 2014.
  • Griewank A, Walther A. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM; 2008.
  • Tuncer İH, Kaya H, Tiftikci H. OTOMATİK TÜREV ARAÇLARI İLE AYRIK ADJOINT ÇÖZÜCÜ GELİŞTİRİLMESİ. 7. Havacılık ve Uzay Konf., Samsun/Türkiye: 2018, p. 1.
  • D. M. W. Powers. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness & correlation 2011. https://doi.org/10.9735/2229-3981.
  • Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86:2278–324. https://doi.org/10.1109/5.726791.
  • Xiao H, Rasul K, Vollgraf R. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms 2017.
  • Zagoruyko S, Komodakis N. Wide Residual Networks 2016.
  • Kingma DP, Ba J. Adam: A Method for Stochastic Optimization 2014.

A NEW ORTOGONAL REGULARIZATION APPROACH FOR DEEP NETWORKS

Year 2024, , 18 - 34, 30.04.2024

Kazım Fırıldak , Gaffari Çelik , Muhammed Fatih Talu

https://doi.org/10.54365/adyumbd.1390894

Abstract

Orthogonal Regularization (OR) is used to prevent overfitting, gradient explosion and Vanishing Gradient in deep networks. OR methods developed for deep learning in the literature aim to learn network weights as unit orthogonal vectors. In this article, the function that enables unit orthogonal learning of network weights for binary groups is added to the cost function. This method is tested on various data sets (artificial data and real data) in artificial neural networks and convolutional neural networks. In addition, the proposed method is compared with methods such as Soft Orthogonal (SO), Double Soft Orthogonal (DSO), Mutual Coherence (MC) and Spectral Restricted Isometry Property (SRIP), which are prominent in the literature, in terms of accuracy, execution time and error rate metrics. As a result of the comparison, an improvement of 1% to 5% is achieved in the accuracy metric in networks using different data sets. The proposed method increases the test success from 92.96 to 93.90% in the Resnet 110 network and from 95.84% to 96.78% in the Resnet 28-10 dataset on the Cifar10 dataset.

Keywords

Orthogonal Regularization Neural Networks Convolutional Neural Networks

References

  • Szegedy C, Wei Liu, Yangqing Jia, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. 2015 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2015, p. 1–9. https://doi.org/10.1109/CVPR.2015.7298594.
  • Krizhevsky A, Hinton GE. ImageNet Classification with Deep Convolutional Neural Networks. Adv Neural Inf Process Syst 2012;1907–1105:1–9.
  • Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial nets. Adv. Neural Inf. Process. Syst., 2014.
  • Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks 2015.
  • Çelik G, Talu MF. Resizing and cleaning of histopathological images using generative adversarial networks. Phys A Stat Mech Its Appl 2019:122652. https://doi.org/10.1016/j.physa.2019.122652.
  • Iizuka O, Kanavati F, Kato K, Rambeau M, Arihiro K, Tsuneki M. Deep Learning Models for Histopathological Classification of Gastric and Colonic Epithelial Tumours. Sci Rep 2020; 10:1504. https://doi.org/10.1038/s41598-020-58467-9.
  • Akram T, Lodhi HMJ, Naqvi SR, Naeem S, Alhaisoni M, Ali M, et al. A multilevel features selection framework for skin lesion classification. Human-Centric Comput Inf Sci 2020;10:12. https://doi.org/10.1186/s13673-020-00216-y.
  • Xu W, Keshmiri S, Wang G. Adversarially Approximated Autoencoder for Image Generation and Manipulation. IEEE Trans Multimed 2019;21:2387–96. https://doi.org/10.1109/TMM.2019.2898777.
  • Turkoglu M, Hanbay D, Sengur A. Multi-model LSTM-based convolutional neural networks for detection of apple diseases and pests. J Ambient Intell Humaniz Comput 2019. https://doi.org/10.1007/s12652-019-01591-w.
  • Grießhaber D, Vu NT, Maucher J. Low-resource text classification using domain-adversarial learning. Comput Speech Lang 2020;62:101056. https://doi.org/10.1016/j.csl.2019.101056.
  • Yin Y, Li H, Fu W. Faster-YOLO: An accurate and faster object detection method. Digit Signal Process 2020;102:102756. https://doi.org/10.1016/j.dsp.2020.102756.
  • Wu Y, Zhang Z, Wang G. Unsupervised Deep Feature Transfer for Low Resolution Image Classification. 2019 IEEE/CVF Int. Conf. Comput. Vis. Work., IEEE; 2019, p. 1065–9. https://doi.org/10.1109/ICCVW.2019.00136.
  • Salimans T, Kingma DP. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Adv. Neural Inf. Process. Syst., 2016.
  • Huang L, Liu X, Lang B, Yu AW, Wang Y, Li B. Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks. 32nd AAAI Conf Artif Intell AAAI 2018 2017.
  • Martin CH, Mahoney MW. Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning 2018.
  • Xu X, Wang G, Sullivan A, Zhang Z. Towards Learning Affine-Invariant Representations via Data-Efficient CNNs 2019.
  • Wang J, Zeng Z, Hou ZG. Advances in neural networks. vol. 149. 2015. https://doi.org/10.1016/j.neucom.2014.07.026.
  • Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Teh YW, Titterington M, editors. Proc. Thirteen. Int. Conf. Artif. Intell. Stat., vol. 9, Chia Laguna Resort, Sardinia, Italy: PMLR; 2010, p. 249–56.
  • He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. 2016 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2016, p. 770–8. https://doi.org/10.1109/CVPR.2016.90.
  • Veit A, Wilber M, Belongie S. Residual Networks Behave Like Ensembles of Relatively Shallow Networks 2016.
  • Mishkin D, Matas J. All you need is a good init 2015.
  • Cui L, Zhai H, Lin H. A Novel Orthogonal Extreme Learning Machine for Regression and Classification Problems. Symmetry (Basel) 2019;11:1284. https://doi.org/10.3390/sym11101284.
  • Shin J, Koo B, Kim Y, Paik J. Deep Binary Classification via Multi-Resolution Network and Stochastic Orthogonality for Subcompact Vehicle Recognition. Sensors 2020;20:2715. https://doi.org/10.3390/s20092715.
  • Jia K, Li S, Wen Y, Liu T, Tao D. Orthogonal Deep Neural Networks. IEEE Trans Pattern Anal Mach Intell 2020:1–1. https://doi.org/10.1109/TPAMI.2019.2948352.
  • Zhang Z, Ma W, Wu Y, Wang G. Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters. 2020 IEEE Winter Conf. Appl. Comput. Vis., IEEE; 2020, p. 1044–8. https://doi.org/10.1109/WACV45572.2020.9093466.
  • Bansal N, Chen X, Wang Z. Can we gain more from orthogonality regularizations in training deep CNNs? Adv Neural Inf Process Syst 2018;2018-Decem:4261–71.
  • Xie D, Xiong J, Pu S. All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation. 2017 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE; 2017, p. 5075–84. https://doi.org/10.1109/CVPR.2017.539.
  • Zhang L, Li D, Guo Q. Deep Learning From Spatio-Temporal Data Using Orthogonal Regularizaion Residual CNN for Air Prediction. IEEE Access 2020;8:66037–47. https://doi.org/10.1109/ACCESS.2020.2985657.
  • Zhu F, Liang Q. OCRNN: An orthogonal constrained recurrent neural network for sleep analysis based on EEG data. Ad Hoc Networks 2020;104:102178. https://doi.org/10.1016/j.adhoc.2020.102178.
  • Zhang J, Yao R, Ge W, Gao J. Orthogonal convolutional neural networks for automatic sleep stage classification based on single-channel EEG. Comput Methods Programs Biomed 2020;183:105089. https://doi.org/10.1016/j.cmpb.2019.105089.
  • Jalwana MAAK, Akhtar N, Bennamoun M, Mian A. Orthogonal Deep Models as Defense Against Black-Box Attacks. IEEE Access 2020;8:119744–57. https://doi.org/10.1109/ACCESS.2020.3005961.
  • Rodríguez P, Gonzàlez J, Cucurull G, Gonfaus JM, Roca X. Regularizing CNNs with Locally Constrained Decorrelations 2016.
  • Balestriero R, richard baraniuk. A Spline Theory of Deep Learning. In: Dy J, Krause A, editors. Proc. 35th Int. Conf. Mach. Learn., vol. 80, PMLR; 2018, p. 374–83.
  • Donoho DL. Compressed sensing. IEEE Trans Inf Theory 2006;52:1289–306. https://doi.org/10.1109/TIT.2006.871582.
  • Lu C, Li H, Lin Z. Optimized Projections for Compressed Sensing via Direct Mutual Coherence Minimization 2015.
  • Yoshida Y, Miyato T. Spectral Norm Regularization for Improving the Generalizability of Deep Learning 2017.
  • Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition 2014.
  • Griewank A, Walther A. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM; 2008.
  • Tuncer İH, Kaya H, Tiftikci H. OTOMATİK TÜREV ARAÇLARI İLE AYRIK ADJOINT ÇÖZÜCÜ GELİŞTİRİLMESİ. 7. Havacılık ve Uzay Konf., Samsun/Türkiye: 2018, p. 1.
  • D. M. W. Powers. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness & correlation 2011. https://doi.org/10.9735/2229-3981.
  • Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86:2278–324. https://doi.org/10.1109/5.726791.
  • Xiao H, Rasul K, Vollgraf R. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms 2017.
  • Zagoruyko S, Komodakis N. Wide Residual Networks 2016.
  • Kingma DP, Ba J. Adam: A Method for Stochastic Optimization 2014.
There are 44 citations in total.

Details

Primary Language Turkish
Subjects Pattern Recognition
Journal Section Makaleler
Authors

Kazım Fırıldak 0000-0002-1958-3627

Gaffari Çelik 0000-0001-5658-9529

Muhammed Fatih Talu 0000-0003-1166-8404

Publication Date April 30, 2024
Submission Date November 14, 2023
Acceptance Date April 22, 2024
Published in Issue Year 2024

Cite

APA Fırıldak, K., Çelik, G., & Talu, M. F. (2024). DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi, 11(22), 18-34. https://doi.org/10.54365/adyumbd.1390894
AMA Fırıldak K, Çelik G, Talu MF. DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi. April 2024;11(22):18-34. doi:10.54365/adyumbd.1390894
Chicago Fırıldak, Kazım, Gaffari Çelik, and Muhammed Fatih Talu. “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi 11, no. 22 (April 2024): 18-34. https://doi.org/10.54365/adyumbd.1390894.
EndNote Fırıldak K, Çelik G, Talu MF (April 1, 2024) DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi 11 22 18–34.
IEEE K. Fırıldak, G. Çelik, and M. F. Talu, “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”, Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi, vol. 11, no. 22, pp. 18–34, 2024, doi: 10.54365/adyumbd.1390894.
ISNAD Fırıldak, Kazım et al. “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi 11/22 (April 2024), 18-34. https://doi.org/10.54365/adyumbd.1390894.
JAMA Fırıldak K, Çelik G, Talu MF. DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi. 2024;11:18–34.
MLA Fırıldak, Kazım et al. “DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI”. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi, vol. 11, no. 22, 2024, pp. 18-34, doi:10.54365/adyumbd.1390894.
Vancouver Fırıldak K, Çelik G, Talu MF. DERİN AĞLAR İÇİN YENİ BİR BİRİMDİK DÜZGÜNLEŞTİRME YAKLAŞIMI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi. 2024;11(22):18-34.