A Comparative Study on Optimization Methods for Residual Neural Network Solutions of Periodic Boundary Value Problems

Öz

This paper investigates using residual neural networks to solve second-order differential equations with periodic boundary conditions. After mentioning the existing numerical methods for ordinary differential equations, approximate solutions of the given equations are obtained with residual neural networks (ResNet). In the training stage of the residual neural model constructed in this process, experimental studies focusing on the Sum of Squared Error (SSE) metrics were conducted to examine their effects on the solution using methods such as AdaDelta, Adam, Nesterov Momentum, and Gradient Descent, which are frequently used in deep learning and derivative-based optimization algorithms. Our findings show that the proposed deep learning model is an effective alternative to traditional numerical solution methods used in numerical analysis.

Anahtar Kelimeler

• Artificial Neural Network
• Deep Learning
• Differential Equations
• Optimization
• Periodic Boundary Conditions

Periyodik Sınır Değer Problemlerinin Artık Sinir Ağı Çözümleri için Optimizasyon Yöntemleri Üzerine Karşılaştırmalı Bir Çalışma

Öz

Bu makale, periyodik sınır koşullarına sahip ikinci mertebeden diferansiyel denklemlerin çözümünde artık sinir ağlarının kullanımını araştırmaktadır. Çalışmada, adi diferansiyel denklemler için mevcut nümerik tekniklerden bahsedildikten sonra, verilen denklemlerin artık sinir ağları (ResNet) ile yaklaşık çözümleri elde edilmiştir. Bu süreçte oluşturulan artık sinir modelinin eğitimi aşamasında, derin öğrenmede sıklıkla kullanılan ve türev tabanlı optimizasyon algoritmalarından olan AdaDelta, Adam, Nesterov Momentum ve Gradient Descent gibi yöntemleri kullanarak çözüme olan etkilerini incelemek için toplam karesel hata (SSE) metriklerine odaklanan deneysel çalışmalar yapılmıştır. Bulgularımız, önerilen derin öğrenme modelinin nümerik analizde kullanılan geleneksel sayısal çözüm yöntemlerine karşı etkili bir alternatif olduğunu ortaya koymaktadır.

Anahtar Kelimeler

• Yapay Sinir Ağları
• Derin Öğrenme
• Diferansiyel Denklemler
• Optimizasyon
• Periyodik Sınır Koşulları

Kaynakça

1. Aftabizadeh, A.R., Xu, J. ve Gupta, C.P. (1990). Periodic boundary value problems for third order ordinary differential equations. Nonlinear Anal., Theory Methods Appl. 14(1), 1-10 (1990). doi: 10.1016/0362-546X(90)90130-9.
2. Agarwal, R., Sun, Y. ve Wong, P. (2010). Existence of positive periodic solutions of periodic boundary value problem for second order ordinary differential equations. Acta Math. Hung. 129(1-2), 166-181. doi: 10.1007/ s10474-010-9268-6.
3. Akila Agnes, S., Anitha, J. (2019). Analyzing the Effect of Optimization Strategies in Deep Convolutional Neural Network. In: Hemanth, J., Balas, V. (eds) Nature Inspired Optimization Techniques for Image Processing Applications. Intelligent Systems Reference Library, vol 150. Springer, Cham. doi: 10.1007/978-3-319-96002-9_10
4. Baslandze, S.R. ve Kiguradze, I.T. (2006). On the unique solvability of a periodic boundary value problem for third-order linear differential equations. Differ. Equ. 42, 165-171. doi: 10.1134/S0012266106020029.
5. Bikchantaev, I.A. (2020). Periodic Boundary Value Problem for a Linear Elliptic Equation of the Second Order in a Half Plane.. Differ. Equ. 56, 813–818.doi: 10.1134/S0012266120070010.
6. Cabada, A. (1995). The method of lower and upper solutions for third order periodic boundary value problems. J. Math. Anal. Appl. 195, 568-589. doi: 10.1006/jmaa.1995.1375.
7. Chu, J. ve Zhou, Z. (2006). Positive solutions for singular non-linear third-order periodic boundary value problems. Nonlinear Anal., Theory Methods Appl. 64(7), 1528-1542. doi: 10.1016/j.na.2005.07.005.
8. El-Sayed, A.M.A. ve Gaafar, F.M. (2018). Existence of Solutions for Singular Second-Order Ordinary Differential Equations with Periodic and Deviated Nonlocal Multipoint Boundary Conditions. J. Funct. Spaces,11. doi: 10.1155/2018/9726475.

9. Fatima, N. (2020). Enhancing Performance of a Deep Neural Network: A Comparative Analysis of Optimization Algorithms. ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, 9(2), 79–90. doi: 10.14201/ADCAIJ2020927990
10. Fu, X. ve Wang, W. (2010). Periodic Boundary Value Problems for Second-Order Functional Differential Equations. J. Inequal. Appl. 2010, 598405. doi: 10.1155/2010/598405.
11. Geng, F., Xu, Y. ve Zhu, D. (2008). Periodic boundary value problems for first-order impulsive dynamic equations on time scales. Nonlinear Anal., Theory Methods Appl. 69(11), 4074-4087. doi: 10.1016/j.na.2007.10.038.
12. Graef, J.R., Kong, L. ve Wang, H. (2008). A periodic boundary value problem with vanishing Green’s function. Appl. Math. Lett. 21(2), 176-180. doi: 10.1016/j.aml.2007.02.019.
13. Günel, K. ve Gör, İ. (2022). Solving Dirichlet boundary problems for ODEs via swarm intelligence. Math. Sci., 16, 325–341. doi: 10.1007/s40096-021-00424-2.
14. Hall, R. (2008). Periodic boundary-value problem for third-order linear functional differential equations. Ukr. Math. J. 60, 481–494 (2008). https://doi.org/10.1007/s11253-008-0069-9.
15. He, Z. ve He, X. (2004). Periodic boundary value problems for first order impulsive integro-differential equations of mixed type, J. Math. Anal. Appl. 296(1), 8-20. doi: 10.1016/j.jmaa.2003.12.047.
16. Hu, S. v Lakshmikantham, V. (1989). Periodic Boundary Value Problems for Second Order Impulsive Differential Systems. Nonlinear Anal., Theory Methods Appl. 13(1),75-85. doi: 10.1016/0362-546X(89)90036-9.
17. İşman, G. (2024). Derin Sinir Ağları Kullanarak Periyodik Sınır Koşullu Diferansiyel Denklemlerin Nümerik Çözümlerinin Elde Edilmesi, Doktora Tezi, Aydın Adnan Menderes Üniversitesi.
18. Kingma, D. P., ve Ba, J. (2014). Adam: A Method for Stochastic Optimization. ArXiv. /abs/1412.6980
19. Kong, L., Wang, S. ve Wang J. (2001). Positive solution of a nonlinear third-order periodic boundary value problem. Int. J. Comput. Appl. Math. 132(2), 247-253. doi: 10.1016/S0377-0427(00)00325-3.
20. Kulikov, A.N. ve Kulikov, D.A. (2021). Cahn–Hilliard equation with two spatial variables. Pattern formation. Theor. Math. Phys. 207, 782–798. doi: 10.1134/S0040577921060088.
21. Lakshmikantham, V. (1989). Periodic Boundary Value Problems of First and Second Order Differential Equations. J. Appl. Math, Simulation, 2(3) 8, doi: 10.1155/S1048953389000110.
22. Liu, Y. (2007). Multiple solutions of periodic boundary value problems for first order differential equations. Comput. Math. Appl. 54(1), 1-8. doi: 10.1016/j.camwa.2006.09.007.
23. Mukhigulashvili, S. (2007). On a periodic boundary value problem for third order linear functional differential equations. Nonlinear Anal., Theory Methods Appl. 66(2), 527-535. doi: 10.1016/j.na.2005.11.046.
24. Nieto, J.J. (2002). Periodic Boundary Value Problems for First-Order Impulsive Ordinary Differential Equations. Nonlinear Anal., Theory Methods Appl, 51(7). 1223-1232. doi: 10.1016/S0362-546X(01)00889-6.
25. Omari, P. ve Trombetta, M. (1992). Remarks on the lower and upper solutions method for second- and third-order periodic boundary value problems. Appl. Math. Comput. 50(1), 1-21. doi: 10.1016/0096-3003(92)90007-N.
26. Sun, J. ve Liu, Y. (2005). Multiple Positive Solutions of Singular Third-Order Periodic Boundary Value Problem. Acta Math. Sci. 25(1), 81-88. doi: 10.1016/S0252-9602(17)30263-1.
27. Taddei, V. ve Zanolin, F. (2007). Bound Sets and Two-Point Boundary Value Problems for Second Order Differential Equations. Georgian Math. J. 14(2), 385-402. doi: 10.1515/GMJ.2007.385.
28. Tian, Y., Zhang, Y., Zhang, H. (2023). Recent Advances in Stochastic Gradient Descent in Deep Learning. Mathematics, 11(3): 682. doi: 10.3390/math11030682.
29. Wang, W., Shen, J. ve Nieto, J.J. (2011). Periodic boundary value problems for second order functional differential equations. J. Appl. Math. Comput. 36, 173–186. doi: 10.1007/s12190-010-0395-6.
30. Wang, Y., Li, J. ve Cai, Z. (2017). Positive solutions of periodic boundary value problems for the second-order differential equation with a parameter. Boundary Value Problems, 49. doi: 10.1186/s13661-017-0776-y.
31. Wang, J., Zhang, W. ve Fečkan, M. (2021). Periodic boundary value problem for second-order differential equations from geophysical fluid flows. Monatsh. Math., 195, 523–540. doi: 10.1007/s00605-021-01539-3.
32. Yu, H. ve Pei, M. (2010). Solvability of a nonlinear third-order periodic boundary value problem. Appl. Math. Lett. 23(8), 892-896. doi: 10.1016/j.aml.2010.04.005.
33. Zhao, Y., Chen, H. ve Qin, B. (2014). Periodic boundary value problems for second-order functional differential equations with impulse. Adv. Differ. Equ. 2014, 134. doi: 10.1186/1687-1847-2014-134.