The Effect of Using Fractional Derivatives in ANNs on Learning

Abstract

During the design phase of artificial neural networks (ANNs), optimization techniques, the number of layers and neurons, as well as activation and loss functions—collectively referred to as hyperparameters—play a decisive role in model performance. Gradient-based optimization methods are widely employed throughout the learning process of these models. However, Stochastic Gradient Descent (SGD) often encounters critical issues such as overfitting, underfitting, poor generalization, and failure to converge. To address these challenges and to achieve more effective modeling of systems through fractional-order derivatives, the KarcıFANN optimization method is employed. In this study, models employing the KarcıFANN and SGD learning algorithms were designed for the classification of the MNIST and Dry Bean datasets. These models were trained using various configurations of layer numbers, as well as different activation and loss functions, and their performances were comparatively evaluated. The results demonstrate that incorporating fractional-order derivatives leads to improved model performance, highlighting the potential of the KarcıFANN method as an effective alternative to the conventional SGD algorithm.

Keywords

• KarcıFANN
• SGD
• Fractional Derivatives
• Classification.

YSA’larda Kesir Dereceli Türev Kullanımının Öğrenmeye Etkisi

Abstract

YSA’ların tasarımı aşamasında kullanılan optimizasyon yöntemleri, katman ve nöron sayıları, aktivasyon ve hata fonksiyonları gibi hiperparametreler, modellerin başarımını belirlemektedir. Gradyan tabanlı optimizasyon yöntemleri, modellerin öğrenme sürecinde yaygın olarak kullanılmaktadır. SGD yönteminde ezberleme, öğrenememe, genelleme yapamama, yakınsayamama gibi önemli problemlerle karşılaşılmaktadır. Bu problemleri çözmek ve kesir dereceli türevlerle sistemleri daha iyi modellemek amacıyla KarcıFANN optimizasyon yöntemi kullanılmaktadır. Bu çalışmada, MNIST ve Dry Bean veri setlerinin sınıflandırılması amacıyla KarcıFANN ve SGD öğrenme yöntemlerinin kullanıldığı modeller tasarlanmıştır. Bu modeller, çeşitli katman sayısı konfigürasyonları ile farklı aktivasyon ve hata fonksiyonları kullanılarak eğitilmiş ve performansları karşılaştırmalı olarak değerlendirilmiştir. Elde edilen bulgular, kesir dereceli türev kullanımıyla modellerin performansının arttığını ve KarcıFANN yönteminin SGD’ye alternatif bir yaklaşım olarak değerlendirilebileceğini göstermektedir.

Keywords

• KarcıFANN
• SGD
• Kesir Dereceli Türev
• sınıflandırma.

References

1. Chen, S., Zhang, C. & Mu, H. An Adaptive Learning Rate Deep Learning Optimizer Using Long and Short-Term Gradients Based on G–L Fractional-Order Derivative. Neural Process Lett 56, 106 (2024). https://doi.org/10.1007/s11063-024-11571-7
2. Cheng W, Pu R, Wang B. AMC: Adaptive Learning Rate Adjustment Based on Model Complexity. Mathematics. 2025; 13(4):650. https://doi.org/10.3390/math13040650
3. Christen, P., Hand, D. J., & Kirielle, N. (2023). A review of the F-measure: its history, properties, criticism, and alternatives. ACM Computing Surveys, 56(3), 1-24.
4. Dogo, E. M., Afolabi, O. J., Nwulu, N. I., Twala, B., & Aigbavboa, C. O. (2018). A comparative analysis of gradient descent-based optimization algorithms on convolutional neural networks. In 2018 international conference on computational techniques, electronics and mechanical systems (CTEMS,) pp. 92-99. IEEE.
5. Dubey, S. R., Singh, S. K., & Chaudhuri, B. B. (2022). Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503, 92-108. doi: 10.1016/j.neucom.2022.06.111.
6. Herrera-Alcántara, O. (2022). Fractional Derivative Gradient-Based Optimizers for Neural Networks and Human Activity Recognition. Applied Sciences, 12(18), 9264. https://doi.org/10.3390/app12189264
7. Jadon, A., Patil, A., & Jadon, S. (2024). A comprehensive survey of regression-based loss functions for time series forecasting. In International Conference on Data Management, Analytics & Innovation (pp. 117-147). Singapore: Springer Nature Singapore.
8. Jeong, J. J., & Koo, G. (2024). AdaLo: Adaptive learning rate optimizer with loss for classification. SSRN. https://doi.org/10.2139/ssrn.4737256

9. Karakurt, M. & İşeri, İ. (2022). Patoloji Görüntülerinin Derin Öğrenme Yöntemleri İle Sınıflandırılması. Avrupa Bilim ve Teknoloji Dergisi, (33), 192-206.
10. Karakurt, M., Saygılı, H. ve Karcı, A. (2024). Comparison of Activation Functions in the KarcıFANN Method. 8th International Artificial Intelligence and Data Processing Symposium (IDAP2024), IEEE, doi: 10.1109/IDAP64064.2024.10711149.
11. Karakurt, M., Saygili, H., & Karci, A. (2025). Karcı fractional artificial neural networks (KarcıFANN): a new artificial neural networks model without learning rate and its problems. Turkish Journal of Electrical Engineering and Computer Sciences, 33(3), 248-263.
12. Karcı, A. (2013a). “A New Approach for Fractional Order Derivative and Its Applications”, Universal Journal of Engineering Sciences, Vol:1, pp: 110-117.
13. Karci, A. (2013b). Generalized fractional order derivatives, its properties and applications. arXiv preprint arXiv:1306.5672.
14. Karcı, A. (2015a). Kesir Dereceli Türevin Yeni Yaklaşımının Özellikleri. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 30(3), 487-501.
15. Karcı, A. (2015b). “Properties of Fractional Order Derivatives for Groups of Relations/Functions”, Universal Journal of Engineering Sciences, vol:3, pp:39-45.
16. Karci, A. (2015c). Chain rule for fractional order derivative. Science Innovation, Vol:3, pp:63-67.
17. Karcı, A. (2019). Properties of Karcı’s Fractional Order Derivative. Universal Journal of Engineering Science, Vol:7, pp:32-38.
18. Karci, A. (2021). Fractional Order Integration: A New Perspective based on Karcı’s Fractional Order Derivative. Computer Science , 6(2), 102-105.
19. Koşan, M. A., Coşkun, A., & Karacan, H. (2019). Yapay zeka yöntemlerinde entropi. Journal of Information Systems and Management Research, 1(1), 15-22.
20. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
21. Lewis, D. D. (1991). Evaluating text categorization. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, 312-138. California, https://aclanthology.org/H91-1061.pdf.
22. McCulloch, W. S. and Pitts, W. 1943. A Logical Calculus of the İdeas İmmanent in Nervous Activity. The Bulletin of Mathematical Biophysics, 5:4, 115-133.
23. Nwankpa, C., Ijomah, W., Gachagan, A., & Marshall, S. (2018). Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378.
24. Powers, D. M. (2020). Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061.
25. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. nature, 323(6088), 533-536.
26. Saygılı, H., Karakurt, M. & Karcı, A. (2024). Comparison of Loss Functions in the KarcıFANN Method. In 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), (pp. 1-5). IEEE.
27. Saygılı, H., Karakurt, M. & Karcı, A. (baskıda). Karcı kesir dereceli yapay sinir ağı (KarcıFANN): öğrenme oranı, aşırı uyum ve yetersiz uyum sorunlarını çözme. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi.
28. Seyyarer, E., Ayata, F., Uçkan, T., & Karci, A. (2020). Derin öğrenmede kullanilan optimizasyon algoritmalarinin uygulanmasi ve kiyaslanmasi. Anatolian Journal of Computer Sciences, 5(2), 90-98.
29. Tian, Y., Zhang, Y., & Zhang, H. (2023). Recent advances in stochastic gradient descent in deep learning. Mathematics, 11(3), 682.
30. Yang, E., Pan, J., Wang, X., Yu, H., Shen, L., Chen, X., Xiao, L., Jiang, J., & Guo, G. (2022). AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning. AAAI Conference on Artificial Intelligence.