Analyzing the Convergence Ability of the KarcıFANN Method

Öz

In gradient descent–based backpropagation algorithms, which are among the most widely used techniques for the optimization of ANNs, the error between the network’s output and the expected output is calculated, and this error is propagated backward to update the weights. The process of updating the weights significantly affects both the learning time and the performance of the model. In gradient descent–based algorithms, choosing a learning rate that is too large or too small may lead to problems such as overfitting, failure to learn, or the weights not converging. During the training phase of ANNs, if the weights follow a stable curve without oscillations as the number of iterations increases, this indicates either successful learning or that the model is stuck at a local minimum. To distinguish between these cases, both the weight change graphs and the error values produced by the network should be examined.In this study, the effects of the KarcıFANN method—which uses fractional derivatives to better represent models and to address the problems encountered in ANNs—on convergence, overfitting, and learning performance were investigated. The KarcıFANN method, in which a fractional derivative is used instead of a fixed learning rate, was compared with ADAM, Momentum-based GD, and SGD methods. To examine the effects of derivatives on learning, the XOR problem was addressed in the experimental studies, and the weight changes of the methods were observed. When the MSE values and weight change graphs were analyzed, it was observed that the most successful method was Momentum-based GD, the second most successful was the KarcıFANN method, and the ADAM method got stuck at a local minimum.

Anahtar Kelimeler

• KarcıFANN
• artificial neural networks
• ADAM
• SGD
• momentum-based GD

KarcıFANN Yönteminin Yakınsama Kabiliyetinin Analiz Edilmesi

Öz

YSA’ların optimizasyonunda yaygın kullanıma sahip tekniklerden gradyan iniş tabanlı geriye yayılım algoritmalarında, ağın çıktısı ile beklenen çıktı arasındaki hata hesaplanmakta ve bu hata geriye doğru yayılarak ağırlıklar güncellenmektedir. Ağırlıkların güncellenmesi işlemi, modelin öğrenme süresini ve performansını önemli ölçüde etkilemektedir. Gradyan iniş tabanlı algoritmalarda kullanılan öğrenme katsayısının çok büyük ya da küçük seçilmesi ezberleme, öğrenememe ve ağırlıkların yakınsamaması gibi problemlere neden olmaktadır. YSA’ların eğitimi aşamasında ağırlıkların, iterasyon sayısı arttıkça salınımlar yapmadan kararlı bir eğri çizmesi, başarılı bir öğrenme gerçekleştiğini ya da yerel minimum bir noktaya takıldığını göstermektedir. Bu ayrımı yapabilmek için ağırlık değişim grafikleriyle beraber ağın ürettiği hata değerlerine bakılmalıdır. Bu makalede, kesir dereceli türevle modelleri daha iyi ifade etmek ve YSA’larda karşılaşılan problemleri çözmek amacıyla kullanılan KarcıFANN yönteminin yakınsama, ezberleme durumu ve öğrenme performansına etkisi incelenmiştir. Sabit bir öğrenme katsayısı yerine kesir dereceli türevin kullanıldığı KarcıFANN yöntemi ile ADAM, Momentumlu GD ve SGD yöntemleri karşılaştırılmıştır. Türevlerin öğrenmeye olan etkilerini incelemek amacıyla XOR probleminin çözümü deneysel çalışmalarda ele alınmış ve yöntemlerin ağırlık değişimleri gözlemlenmiştir. MSE değerleri ve ağırlık değişim grafikleri incelendiğinde en başarılı yöntemin Momentumlu GD, ikinci başarılı yöntemin KarcıFANN yöntemi olduğu ve ADAM yönteminin de yerel bir minimum noktaya takıldığı görülmektedir.

Anahtar Kelimeler

• KarcıFANN
• yapay sinir ağları
• ADAM
• SGD
• momentumlu GD

Kaynakça

1. Abraham A. Meta-Learning Evolutionary Artificial Neural Networks, Neurocomputing Journal, 2004, Vol. 56c, Elsevier Science, Netherlands, (1–38).
2. Rumelhart DE, Durbin R, Golden RM, Chauvin Y. Backpropagation: the basic theory, 1995.
3. Smith LN. A disciplined approach to neural network hyper-parameters: Part 1 – Learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820, 2018.
4. Brutzkus A, Globerson A. Why do larger models generalize better? A theoretical perspective via the XOR problem. Proceedings of the 36th International Conference on Machine Learning (ICML), 2019, 97, 822–830.
5. Pinto RC, Tavares AR. (2024, September 17). PReLU: Yet Another Single-Layer Solution to the XOR Problem (arXiv:2409.10821v1) [Preprint].
6. Yang C, Kim H, Adhikari SP, Chua LO. A Circuit-Based Neural Network with Hybrid Learning of Backpropagation and Random Weight Change Algorithms. Sensors. 2017, 17(1):16.
7. Starodub A, Eliseeva N, Georgiev M. Gradient-based algorithm for tracking the activity of neural network weights changing. EPJ Web of Conferences, 248, 2021, 01012.
8. Li Jun & Diao, Yongfeng & Li, Mingdong & Yin, Xing., Stability Analysis of Discrete Hopfield Neural Networks with the Nonnegative Definite Monotone Increasing Weight Function Matrix. Discrete Dynamics in Nature and Society. 2009.

9. Nagel M, Fournarakis M, Bondarenko Y, Blankevoort T. Overcoming Oscillations in Quantization-Aware Training. International Conference on Machine Learning (ICML),2022,
10. Nagel M, van Baalen M, Blankevoort T, and Welling, M. Data-free quantization through weight equalization and bias correction. In International Conference on Computer Vision (ICCV).
11. Banner R, Nahshan Y, Soudry D. Post training 4-bit quantization of convolutional networks for rapiddeployment. In Advances in Neural Information Processing Systems (NeuRIPS), arXiv:1810.05723v3 [cs.CV] 29 May 2019.
12. Agrawal AM, Tendle A. (2020). Investigating learning in deep neural networks using layer-wise weight change [Conference paper]. Proceedings of the International Joint Conference on Neural Networks (IJCNN), 1-8., 2020, IEEE.
13. Wei JL, Wu GC, Liu BQ. et al. An optimal neural network design for fractional deep learning of logistic growth. Neural Comput & Applic 35, 10837–10846 (2023).
14. Bottou L. Stochastic gradient learning in neural networks. Proceedings of Neuro-Nımes, 1991, 91(8), 12.
15. Kingma DP, Ba J, Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR), Banff, 14-16 April 2014.
16. Fu J, Wang B, Zhang H, Zhang , Chen W, Zheng N, When and why momentum accelerates sgd: An empirical study, 2023, arXiv preprint arXiv:2306.09000.
17. Seyyarer E, Ayata F, Uçkan T, Karcı A. Applıcatıons and comparıson of optımızatıon algorıthms used ın deep learnıng. Anatolian Science , 2020, vol.5, no.2, 90-98.
18. Karcı A. A New Approach for Fractional Order Derivative and Its Applications, Universal Journal of Engineering Sciences, 2013, Vol:1, pp: 110-117.
19. Karcı A. Properties of Fractional Order Derivatives for Groups of Relations/Functions, Universal Journal of Engineering Sciences, 2015a , vol:3, pp:39-45.
20. Karcı A. The Properties of New Approach of Fractional Order Derivative, Journal of the Faculty of Engineering and Architecture of Gazi University, 2015b ,Vol.30, pp:487-501.
21. Karci A. Chain rule for fractional order derivative, Science Innovation, 2015c, Vol:3, pp:63-67.
22. Karcı A. Properties of Karcı’s Fractional Order Derivative, Universal Journal of Engineering Science, 2019, Vol:7, pp:32-38.
23. Karci A. Fractional Order Integration: A New Perspective based on Karcı’s Fractional Order Derivative. Computer Science, 2019, 6(2), 102-105.
24. Karakurt M, Saygılı H, Karcı A. RMSE Yönteminde Aktivasyon Fonksiyonlarının Karşılaştırılması, 8th International Artificial Intelligence and Data Processing Symposium (IDAP2024), IEEE.
25. Karakurt M, Saygılı H, Karcı A. Karcı fractional artificial neural networks (KarcıFANN): a new artificial neural networks model without learning rate and its problems. Turkish Journal of Electrical Engineering and Computer Sciences, 33(3), 2025; 248-263.
26. Saygılı H, Karakurt M, Karcı A. Comparison of Loss Functions in the KarcıFANN Method. In 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), 2024 ;(pp. 1-5). IEEE.