The Effect of Weight Decay in the KarcıFANN Method

Abstract

In the modeling of artificial neural networks (ANNs), various regularization methods are employed in conjunction with hyperparameters such as optimization methods, activation functions, and loss functions. Among these methods, weight decay is applied to address critical issues such as gradient explosion and overfitting, which arise from excessive growth of weight vectors during training, and to enhance the generalization performance of models. In this study, the effects of the weight decay hyperparameter on the performance of the KarcıFANN method are analyzed. For this purpose, weight decay was applied to various models designed using the KarcıFANN method. The empirical results derived from the classification on the MNIST and Dry Bean datasets demonstrate that weight decay hyperparameter markedly improves the predictive performance and the generalization capacity of the models.

Keywords

• KarcıFANN
• classification
• regularization methods
• weight decay

KarcıFANN Yönteminde Ağırlık Sönümlemenin Etkisi

Abstract

Yapay sinir ağlarının (YSA) modellenmesinde, optimizasyon yöntemi, aktivasyon ve hata fonksiyonu gibi hiperparametrelerle birlikte çeşitli düzenleme (düzenlileştirme) yöntemleri kullanılmaktadır. Bu yöntemlerden biri olan ağırlık sönümleme işlemi, modellerin eğitimi aşamasında ağırlık vektörlerinin çok fazla büyümesiyle ortaya çıkan gradyan patlaması ve ezberleme gibi önemli problemlerin çözülmesi ve modellerin genelleme performanslarının artırılması amacıyla uygulanmaktadır. Bu çalışmada, ağırlık sönümleme hiperparametresinin, KarcıFANN yönteminin performansına etkileri analiz edilmektedir. Bu amaçla, KarcıFANN yöntemi ile tasarlanan çeşitli modellere ağırlık sönümleme işlemi uygulanmıştır. MNIST ve Dry Bean veri setlerinin sınıflandırılması sonucu elde edilen bulgular, ağırlık sönümleme hiperparametresinin, modellerin başarımını ve genelleme kabiliyetini önemli ölçüde iyileştirdiğini göstermiştir.

Keywords

• KarcıFANN
• sınıflandırma
• düzenleme yöntemleri
• ağırlık sönümleme

References

1. Christen, P., Hand, D. J., & Kirielle, N. (2023). A review of the F-measure: its history, properties, criticism, and alternatives. ACM Computing Surveys, 56(3), 1-24.
2. Galanti, T., Siegel, Z. S., Gupte, A., & Poggio, T. A. (2025, March). SGD with weight decay secretly minimizes the ranks of your neural networks. In The Second Conference on Parsimony and Learning (CPAL 2025 Proceedings Track).
3. Jadon, A., Patil, A., & Jadon, S. (2024). A comprehensive survey of regression-based loss functions for time series forecasting. In International Conference on Data Management, Analytics & Innovation (pp. 117-147). Singapore: Springer Nature Singapore.
4. Karakurt, M., Hark, H., Erdoğan, M. C., & Karci, A. (2022). Karcı Sinir Ağlarının Uygulaması ve Performans Analizi. Computer Science, 7(2), 68-80.
5. Karakurt, M., Saygılı, H. ve Karcı, A. (2024). Comparison of Activation Functions in the KarcıFANN Method. 8th International Artificial Intelligence and Data Processing Symposium (IDAP2024), IEEE, doi: 10.1109/IDAP64064.2024.10711149.
6. Karakurt, M., Saygili, H., & Karci, A. (2025). Karcı fractional artificial neural networks (KarcıFANN): a new artificial neural networks model without learning rate and its problems. Turkish Journal of Electrical Engineering and Computer Sciences, 33(3), 248-263.
7. Karcı, A. (2013a). “A New Approach for Fractional Order Derivative and Its Applications”, Universal Journal of Engineering Sciences, Vol:1, pp: 110-117.
8. Karci, A. (2013b). Generalized fractional order derivatives, its properties and applications. arXiv preprint arXiv:1306.5672.

9. Karcı, A. (2015a). Kesir Dereceli Türevin Yeni Yaklaşımının Özellikleri. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 30(3), 487-501.
10. Karcı, A. (2015b). “Properties of Fractional Order Derivatives for Groups of Relations/Functions”, Universal Journal of Engineering Sciences, vol:3, pp:39-45.
11. Karci, A. (2015c). Chain rule for fractional order derivative. Science Innovation, Vol:3, pp:63-67.
12. Karcı, A. (2019). Properties of Karcı’s Fractional Order Derivative. Universal Journal of Engineering Science, Vol:7, pp:32-38.
13. Karci, A. (2021). Fractional Order Integration: A New Perspective based on Karcı’s Fractional Order Derivative. Computer Science, 6(2), 102-105.
14. Koşan, M. A., Coşkun, A., & Karacan, H. (2019). Yapay zeka yöntemlerinde entropi. Journal of Information Systems and Management Research, 1(1), 15-22.
15. Kosson, P., Messmer, P., & Jaggi, M. (2023). Rotational equilibrium: How weight decay balances learning across neural networks. arXiv preprint arXiv:2305.17212.
16. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
17. Lewis, D. D. (1991). Evaluating text categorization. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, 312-138. California, https://aclanthology.org/H91-1061.pdf.
18. Li, Z., Lyu, K., & Arora, S. (2020). Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33, 14544-14555.
19. Loshchilov, I., & Hutter, F. (2017). Decoupled weight decay regularization. International Conference on Learning Representations 2019. arXiv preprint arXiv:1711.05101.
20. Outmezguine, N. J., & Levi, N. (2024). Decoupled weight decay for any p norm. arXiv preprint arXiv:2404.10824.
21. Powers, D. M. (2020). Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061.
22. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. nature, 323(6088), 533-536.
23. Saygılı, H., Karakurt, M. & Karcı, A. (2024). Comparison of Loss Functions in the KarcıFANN Method. In 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), (pp. 1-5). IEEE.
24. Saygılı, H., Karakurt, M. & Karcı, A. (baskıda). Karcı kesir dereceli yapay sinir ağı (KarcıFANN): öğrenme oranı, aşırı uyum ve yetersiz uyum sorunlarını çözme. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi.
25. Seyyarer, E., Ayata, F., Uçkan, T., & Karci, A. (2020). Derin öğrenmede kullanilan optimizasyon algoritmalarinin uygulanmasi ve kiyaslanmasi. Anatolian Journal of Computer Sciences, 5(2), 90-98.
26. Xie, Z., Sato, I., & Sugiyama, M. (2020). Understanding and scheduling weight decay. Retrieved from https://openreview.net/forum?id=J7V_4aauV6B
27. Van Laarhoven, T. (2017). L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350. https://doi.org/10.48550/arXiv.1706.05350
28. Zhang, G., Wang, C., Xu, B., & Grosse, R. (2018). Three Mechanisms of Weight Decay Regularization. arXiv preprint arXiv:1810.12281.