A THEORETICAL INVESTIGATION ON TRAINING OF PIPE-LIKE NEURAL NETWORK BENCHMARK ARCHITECTURES AND PERFORMANCE COMPARISONS OF POPULAR TRAINING ALGORITHMS

Abstract

Architectures of neural networks affect the training performance of artificial neural networks. For more consistent performance evaluation of training algorithms, hard-to-train benchmarking architectures should be used. This study introduces a benchmark neural network architecture, which is called pipe-like architecture, and presents training performance analyses for popular Neural Network Backpropagation Algorithms (NNBA) and well-known Metaheuristic Search Algorithms (MSA). The pipe-like neural architectures essentially resemble an elongated fraction of a deep neural network and form a narrowed long bottleneck for the learning process. Therefore, they can significantly complicate the training process by causing the gradient vanishing problems and large training delays in backward propagation of parameter updates throughout the elongated pipe-like network. The training difficulties of pipe-like architectures are theoretically demonstrated in this study by considering the upper bound of weight updates according to an aggregated one-neuron learning channels conjecture. These analyses also contribute to Baldi et al.'s learning channel theorem of neural networks in a practical aspect. The training experiments for popular NNBA and MSA algorithms were conducted on the pipe-like benchmark architecture by using a biological dataset. Moreover, a Normalized Overall Performance Scoring (NOPS) was performed for the criterion-based assessment of overall performance of training algorithms.

Keywords

• Artificial Neural Networks
• Network Architectures
• Training Performance
• Backpropagation Algorithms
• Metaheuristic Training

BORU-BENZERİ YAPAY SİNİR AĞI KARŞILAŞTIRMA MİMARİLERİNİN EĞİTİMİ HAKKINDA BİR TEORİK ARAŞTIRMA VE POPULAR EĞİTİM ALGORİTMALARIN PERFORMANS KARŞILAŞTIRILMALARI

Abstract

Sinir ağlarının mimarileri, yapay sinir ağlarının eğitim performansını etkiler. Eğitim algoritmalarının daha tutarlı performans değerlendirmesi için eğitimi zor kıyaslama mimarileri kullanılmalıdır. Bu çalışma, boru-benzeri mimari olarak adlandırılan bir referans sinir ağı mimarisini tanıtmakta ve popüler Sinir Ağı Geriyeyayılım Algoritmaları (SAGA) ve iyi bilinen Metasezgisel Arama Algoritmalarının (MAA) eğitim performansı analizlerini sunmaktadır. Boru-benzeri sinir mimarileri, temelde bir derin sinir ağının uzunlamasına bir kesitini temsil eder ve öğrenme süreci için bir daraltılmış uzun darboğaz oluşturur. Bu nedenle, uzun boru-benzeri ağ boyunca parametre güncellemelerinin geriye doğru yayılmasında gradyan kaybolma problemleri ve büyük eğitim gecikmelerine neden olarak eğitim sürecini önemli ölçüde zorlaştırır. Bu çalışmada boru-benzeri mimarilerin eğitim zorlukları birleştirilmiş tek-nöron öğrenme kanalları konjektörüne göre ağırlık güncellemelerinin üst sınırı dikkate alınarak teorik olarak gösterilmiştir. Bu analizler aynı zamanda Baldi ve arkadaşlarının sinir ağlarının öğrenme kanalı teoremine pratik açıdan da katkıda bulunmaktadır. Popüler NNBA ve MSA algoritmalarının eğitim deneyleri, bir biyolojik veri seti kullanılarak boru benzeri kıyaslama mimarisinde gerçekleştirmiştir. Ayrıca, eğitim algoritmalarının genel performansının ölçüt tabanlı değerlendirmesi için Normalleştirilmiş Genel Performans Puanlaması (NGPP) uygulanmıştır.

Keywords

• Yapay Sinir Ağları
• Ağ Mimarileri
• Eğitim Performansı
• Geriyeyayılım Algoritmaları
• Metasezgisel Eğitim

References

1. Aliev, R.A., Fazlollahi, B., Guirimov, B.G., Aliev, R.R., 2008. Recurrent Fuzzy Neural Networks and Their Performance Analysis. in: Recurr. Neural Networks, InTech. https://doi.org/10.5772/5540.
2. Arifovic, J., Gençay, R., 2001. Using genetic algorithms to select architecture of a feedforward artificial neural network. Phys. A Stat. Mech. Its Appl., 289:574–594. https://doi.org/10.1016/S0378-4371(00)00479-9.
3. Awolusi, T.F., Oke, O.L., Akinkurolere, O.O., Sojobi, A.O., Aluko, O.G., 2019. Performance comparison of neural network training algorithms in the modeling properties of steel fiber reinforced concrete. Heliyon 5:e01115. https://doi.org/10.1016/j.heliyon.2018.e01115.
4. Bahrami, M., Akbari, M., Bagherzadeh, S.A., Karimipour, A., Afrand, M., Goodarzi, M., 2019. Develop 24 dissimilar ANNs by suitable architectures & training algorithms via sensitivity analysis to better statistical presentation: Measure MSEs between targets & ANN for Fe–CuO/Eg–Water nanofluid. Phys. A Stat. Mech. Its Appl. 519:159–168. https://doi.org/10.1016/j.physa.2018.12.031.
5. Bala, J.W., Analytics, D., Bloedorn, E., Bratko, I., 1992. The MONK’s Problems A Performance Comparison of Different Learning Algorithms. http://robots.stanford.edu/papers/thrun.MONK.html Accessed 05 August 2021.
6. Battiti, R., 1992. First- and Second-Order Methods for Learning: Between Steepest Descent and Newton’s Method. Neural Comput., 4:141–166. https://doi.org/10.1162/neco.1992.4.2.141.
7. Beale, E.M.L., 1972. A derivation of conjugate gradients. in F.A. Lootsma, Ed., Numerical methods for nonlinear optimization, Academic Press, London, 39-43.
8. Birattari, M., Kacprzyk, J., 2009. Tuning metaheuristics: a machine learning perspective, Springer, Berlin.

9. Can, A., Dagdelenler, G., Ercanoglu, M., Sonmez, H., 2019. Landslide susceptibility mapping at Ovacık-Karabük (Turkey) using different artificial neural network models: comparison of training algorithms. Bull. Eng. Geol. Environ., 78:89–102. https://doi.org/10.1007/s10064-017-1034-3.
10. Caruana, R., Niculescu-Mizil, A., 2006. An empirical comparison of supervised learning algorithms. ACM Int. Conf. Proceeding Ser., 148:161–168. https://doi.org/10.1145/1143844.1143865.
11. Che, Z.G., Chiang, T.A., Che, Z.H., 2011. Feed-forward neural networks training: a comparison between genetic algorithm and back-propagation learning algorithm. International Journal of Innovative Computing Information and Control , 7(10), 5839-5850.
12. Chen, Z., Ashkezari, A.Z., Tlili, I., 2020. Applying artificial neural network and curve fitting method to predict the viscosity of SAE50/MWCNTs-TiO2 hybrid nanolubricant. Phys. A Stat. Mech. Its Appl., 549:123946. https://doi.org/10.1016/j.physa.2019.123946.
13. Chopard, B., Tomassini, M., 2018. Performance and limitations of metaheuristics. in: Nat. Comput. Ser., Springer Verlag, 191–203. https://doi.org/10.1007/978-3-319-93073-2_11.
14. Coleman, C., Narayanan, D., Kang, D., Zhao, T., Zhang, J., Nardi, L., Bailis, P., Olukotun, K., Zaharia, C.M., 2017. DawnBench: An end-to-end deep learning benchmark and competition, Training. In NIPS ML Systems Workshop.
15. Cömert, Z., Kocamaz, A., 2017. A Study of Artificial Neural Network Training Algorithms for Classification of Cardiotocography Signals. Bitlis Eren Univ. J. Sci. Technol., 7 , 93–103. https://doi.org/10.17678/beuscitech.338085.
16. Csaji, B.C., 2001. Approximation with Artificial Neural networks, Faculty of Science; Eötvö Lorand University, Hungary.
17. Dennis, J.E., Schnabel, R.B., 1996. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Society for Industrial and Applied Mathematics, SIAM. https://doi.org/10.1137/1.9781611971200.
18. Deng, L., Yu, D., 2013. Deep learning: Methods and applications, Found. Trends Signal Process. 7:197–387. https://doi.org/10.1561/2000000039.
19. Ding, S., Li, H., Su, C., et al., 2013. Evolutionary artificial neural networks: a review. Artificial Intelligence Review, 39:251–260. https://doi.org/10.1007/s10462-011-9270-6.
20. Faris, H., Aljarah, I., Al-Betar, M.A., Mirjalili, S., 2018. Grey wolf optimizer: a review of recent variants and applications. Neural Comput. Appl., 30:413–435. https://doi.org/10.1007/s00521-017-3272-5.
21. Fletcher, R., 1964. Function minimization by conjugate gradients. Comput. J., 7:149–154. https://doi.org/10.1093/comjnl/7.2.149.
22. Floreano, D., Dürr, P., Mattiussi, C., 2008. Neuroevolution: from architectures to learning. Evolutionary intelligence, 1(1), 47-62.
23. Fong, S., Deb, S., Yang, X.S., 2018. How meta-heuristic algorithms contribute to deep learning in the hype of big data analytics. Adv. Intell. Syst. Comput., 518:3–25. https://doi.org/10.1007/978-981-10-3373-5_1.
24. Galván, E., Mooney, P., 2021. Neuroevolution in deep neural networks: Current trends and future challenges. IEEE Transactions on Artificial Intelligence, 2: 476-493. https://doi.org/10.1109/TAI.2021.3067574.
25. Ghasemiyeh, R., Moghdani, R.., Sana, S.S., 2017. A Hybrid Artificial Neural Network with Metaheuristic Algorithms for Predicting Stock Price. Cybern. Syst., 48:365–392. https://doi.org/10.1080/01969722.2017.1285162.
26. Glorot, X., Bengio, Y., 2010. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256.
27. Gogna, A.., Tayal, A., 2013. Metaheuristics: Review and application. J. Exp. Theor. Artif. Intell., 25:503–526. https://doi.org/10.1080/0952813X.2013.782347.
28. Goh, C.H., Tung, Y.C.A., Cheng, C.H., 1996. A revised weighted sum decision model for robot selection. Comput. Ind. Eng., 30:193–199. https://doi.org/10.1016/0360-8352(95)00167-0.
29. Goodfellow, I., Bengio, Y., Courville, A., 2016. Deep Learning, MIT Press.
30. Gudise, V.G., Venayagamoorthy, G.K., 2003. Comparison of particle swarm optimization and backpropagation as training algorithms for neural networks. in: 2003 IEEE Swarm Intell. Symp. SIS 2003 - Proc., Institute of Electrical and Electronics Engineers Inc., 2003:110–117. https://doi.org/10.1109/SIS.2003.1202255.
31. Gunantara, N., Nurweda, Putra I.D.N., 2019. The Characteristics of Metaheuristic Method in Selection of Path Pairs on Multicriteria Ad Hoc Networks. J. Comput. Networks Commun., 2019:7983583. https://doi.org/10.1155/2019/7983583.
32. Hagan, M.T., Demuth, H.B., Beale, M.H., 1996. Neural Network Design, Boston, MA: PWS Publishing.
33. Hagan, M.T., Menhaj, M.B., 1994. Training Feedforward Networks with the Marquardt Algorithm. IEEE Trans. Neural Networks., 5:989–993. https://doi.org/10.1109/72.329697.
34. Hinton, G.E., Salakhutdinov, R.R., 2006. Reducing the dimensionality of data with neural networks. Science 313:504–507. https://doi.org/10.1126/science.1127647.
35. Hornik, K., 1991. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4:251–257. https://doi.org/10.1016/0893-6080(91)90009-T.
36. Igel, C., 2014. No free lunch theorems: Limitations and perspectives of metaheuristics. In Theory and principled methods for the design of metaheuristics. Springer, Berlin.
37. Ince, T., Kiranyaz, S., Pulkkinen, J., Gabbouj, M., 2010. Evaluation of global and local training techniques over feed-forward neural network architecture spaces for computer-aided medical diagnosis. Expert Syst. Appl., 37:8450–8461. https://doi.org/10.1016/j.eswa.2010.05.033.
38. Isik, P.X., Sadowski, P., 2016. A theory of local learning, the learning channel. and the optimality of backpropagation. Neural Netw., 83:51-74. https://doi.org/10.1016/j.neunet.2016.07.006
39. Karim, H., Niakan, S.R., Safdari, R., 2018. Comparison of neural network training algorithms for classification of heart diseases. IAES Int. J. Artif. Intell. , 7:185–189. https://doi.org/10.11591/ijai.v7.i4.pp185-189.
40. Kim, P., 2017. Matlab Deep Learning With Machine Learning. Neural Networks and Artificial Intelligence, Apress.
41. Kratsios, A., Bilkoptov, E., 2020. Non-Euclidean Universal Approximation. arXiv preprint arXiv:2006.02341.
42. Manoharan, S., Sathesh, A., 2020. Population Based Meta Heuristics Algorithm for Performance Improvement of Feed Forward Neural Network. Journal of Soft Computing Paradigm, 2(1), 36-46. https://doi.org/10.36548/jscp.2020.1.004.
43. Marquardt, D.W., 1963. An Algorithm for Least-Squares Estimation of Nonlinear Parameters. J. Soc. Ind. Appl. Math. 11:431–441. https://doi.org/10.1137/0111030.
44. Martens, J. , 2010. Deep learning via hessian-free optimization. in ICML, 27:735-742.
45. Mhaskar, H., Liao, Q., Poggio, T., 2016. Learning Functions: When Is Deep Better Than Shallow. arXiv preprint arXiv:1603.00988.
46. Mirjalili, S., Mirjalili, S.M., 2014. A. Lewis, Grey Wolf Optimizer. Adv. Eng. Softw., 69:46–61. https://doi.org/10.1016/j.advengsoft.2013.12.007.
47. Michalewicz, Z., 1992. Genetic algorithm + data structures = evolutionary programs. Springer-Verlag, New York. Melanie, M., 1996. An Introduction to Genetic Algorithms. MIT Press, Cambridge.
48. Moller, M.F., 1993. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks , 1993:6525–533. https://doi.org/10.1016/S0893-6080(05)80056-5.
49. Mosavi, M.R., Khishe, M., Ghamgosar, A., 2016. Classification Of Sonar Data Set Using Neural Network Trained By Gray Wolf Optimization. Neural Netw. World. 26:393-415 https://doi.org/10.14311/nnw.2016.26.023.
50. Oostwal, E., Straat, M., Biehl, M., 2019. Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation. Phys. A Stat. Mech. Its Appl, 564:125517. https://doi.org/10.1016/j.physa.2020.125517.
51. Pan, X., Lee, B., Zhang, C., 2013. A comparison of neural network backpropagation algorithms for electricity load forecasting. In 2013 IEEE International Workshop on Inteligent Energy Systems (IWIES), 22-27.
52. Parejo, J.A., Ruiz-Cortés, A., Lozano, S., Fernandez, P., 2012. Metaheuristic optimization frameworks: A survey and benchmarking. Soft Comput., 16:527–561. https://doi.org/10.1007/s00500-011-0754-8
53. Powell, M.J.D., 1977. Restart procedures for the conjugate gradient method. Math. Program, 12:241–254. https://doi.org/10.1007/BF01593790.
54. Riedmiller, M., Braun, H., 1993. Direct adaptive method for faster backpropagation learning: The RPROP algorithm. in: 1993 IEEE Int. Conf. Neural Networks, Publ by IEEE, 586–591. https://doi.org/10.1109/icnn.1993.298623.
55. Roodschild, M., Sardiñas, J. G., Will, A., 2020. A new approach for the vanishing gradient problem on sigmoid activation. Progress in Artificial Intelligence, 9(4), 351-360.
56. Rusiecki, A., 2012. Robust learning algorithm based on iterative least median of squares. Neural Process. Lett., 36:145–160. https://doi.org/10.1007/s11063-012-9227-z.
57. Scales, L.E., 1985. Introduction to Non-Linear Optimization. Springer-Verlag, New York.
58. Sexton, R.S., Gupta, J.N.D., 2000. Comparative evaluation of genetic algorithm and backpropagation for training neural networks. Inf. Sci. , 129:45–59. https://doi.org/10.1016/S0020-0255(00)00068-2.
59. Sewak, M., Sahay, S.K., Rathore, H., 2018. Comparison of deep learning and the classical machine learning algorithm for the malware detection, in: Proc. - 2018 IEEE/ACIS 19th Int. Conf. Softw. Eng. Artif. Intell. Netw. Parallel/Distributed Comput. SNPD 2018, Institute of Electrical and Electronics Engineers Inc., pp.293–296. https://doi.org/10.1109/SNPD.2018.8441123.
60. Schmidhuber, J., 2015. Deep Learning in neural networks: An overview. Neural Networks 61:85–117. https://doi.org/10.1016/j.neunet.2014.09.003.
61. Shrestha, A., Mahmood, A., 2019. Review of deep learning algorithms and architectures. IEEE Access 7:53040-53065. https://doi.org/10.1109/ACCESS.2019.2912200
62. Stanley, K.O., Clune, J., Lehman, J., Miikkulainen, R., 2019. Designing neural networks through neuroevolution. Nature Machine Intelligence, 1(1), 24-35.
63. Stanujkic, D., Zavadskas, E.K., 2015. A modified Weighted Sum method based on the decision-maker’s preferred levels of performances. Stud. Informatics Control. 24:461-469. https://doi.org/10.24846/v24i4y201510.
64. Strang, G., 2018. The functions of deep learning. SIAM news., 51:1-4.
65. Suganuma, M., Shirakawa, S., Nagao, T., 2017. A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference on - GECCO ’17. ACM Press, New York, New York, USA, 497–504.
66. Sussillo, D., Abbott, L.F., 2014. Random Walk Initialization for Training Very Deep Feedforward Networks. arXiv preprint arXiv:1412.6558. http://arxiv.org/abs/1412.6558 (accessed June 11, 2021).
67. Tagluk, M.E., Isik, I., 2019. Communication in nano devices: Electronic based biophysical model of a neuron. Nano Commun. Netw. , 19:134–147. https://doi.org/10.1016/j.nancom.2019.01.006.
68. Thakkar, A., Mungra, D., Agrawal, A., 2020. Sentiment analysis: An empirical comparison between various training algorithms for artificial neural network, Int. J. Innov. Comput. Appl., 11:9–29. https://doi.org/10.1504/IJICA.2020.105315.
69. Winkler, D.A., Le, T.C., 2017. Performance of Deep and Shallow Neural Networks, the Universal Approximation Theorem. Activity Cliffs, and QSAR, Mol. Inform,36. https://doi.org/10.1002/minf.201600118.
70. Wong, W.K., Ming, C.I., 2019. A Review on Metaheuristic Algorithms: Recent Trends, Benchmarking and Applications, in: 2019 7th Int. Conf. Smart Comput. Commun. ICSCC 2019, Institute of Electrical and Electronics Engineers Inc., 1-5. https://doi.org/10.1109/ICSCC.2019.8843624.
71. Vogl, T.P., Mangis, J.K., Rigler, A.K., Zink, W.T., Alkon, D.L., 1988. Accelerating the convergence of the back-propagation method. Biol. Cybern., 59:257–263. https://doi.org/10.1007/BF00332914.
72. Zamri, N.B.A., Bhuvaneswari, T., Aziz, N.A.B.A., Aziz, N.H.B.A., 2018. Feature selection using simulated Kalman filter (SKF) for prediction of body fat percentage. In Proceedings of the 2018 International Conference on Mathematics and Statistics, 23–27. https://doi.org/10.1145/3274250.3274264.
73. Zeugmann, T., Poupart, P., Kennedy, J., Jin, X., Han, J., Saitta, L., Sebag, M., Peters, J., Bagnell, J.A., Daelemans, W., Webb, G.I., Ting, K.M., Ting, K.M., Webb, G.I., Shirabad, J.S., Fürnkranz, J., Hüllermeier, E., Matwin, S., Sakakibara, Y., Flener, P., Schmid, U., Procopiuc, C.M., Lachiche, N., Fürnkranz, J., 2011. Particle Swarm Optimization. in: Encycl. Mach. Learn., Springer US, Boston, MA, 760–766. https://doi.org/10.1007/978-0-387-30164-8_630.
74. Zhao, X., Xia, L., Zhang, J., Song, W., 2020. Artificial neural network based modeling on unidirectional and bidirectional pedestrian flow at straight corridors. Phys. A Stat. Mech. Its Appl, 547:123825. https://doi.org/10.1016/j.physa.2019.123825.
75. Zhao, Z., Xin, H., Ren, Y., Guo, X., 2010. Application and comparison of BP neural network algorithm in MATLAB, in: 2010 Int. Conf. Meas. Technol. Mechatronics Autom. ICMTMA, 2010: 590–593. https://doi.org/10.1109/ICMTMA.2010.492.
76. Zhu, H., Akrout, M., Zheng, B., Pelegris, A., Jayarajan, A., Phanishayee, A., Schroeder, B., Pekhimenko, G., 2018. Benchmarking and Analyzing Deep Neural Network Training, in: 2018 IEEE Int. Symp. Workload Charact. IISWC 2018, Institute of Electrical and Electronics Engineers Inc., 2018:88–100. https://doi.org/10.1109/IISWC.2018.8573476.