Konvolüsyonel Sinir Ağlarında Hiper-Parametre Optimizasyonu Yöntemlerinin İncelenmesi

Abstract

Konvolüsyonel Sinir Ağları (KSA), katmanlarının en az bir tanesinde matris çarpımı yerine konvolüsyon işleminin kullanıldığı çok katmanlı yapay sinir ağlarının bir türüdür. Özellikle bilgisayarlı görü çalışmalarında çok başarılı sonuçlar elde edilse de KSA hala birçok zorluk içermektedir. Daha başarılı sonuçlar elde etmek için geliştirilen mimarilerin giderek daha derinleşmesi ve kullanılan görüntülerin giderek daha yüksek kalitede olmasıyla daha fazla hesaplama maliyetleri ortaya çıkmaktadır. Hem bu hesaplama maliyetlerinin düşürülmesi, hem de başarılı sonuçlar elde edilebilmesi, güçlü donanımların kullanılmasına ve kurulan ağın parametrelerinin, başka bir deyişle hiper-parametrelerin optimize edilmesine bağlıdır. Yaptığımız bu çalışmada, KSA hiper-parametrelerinin optimize edilmesi için yaygın olarak kullanılan yöntemleri, optimize edilen hiper-parametreleri, bu parametreler için tanımlanan değer aralıklarını, veri setlerini ve elde edilen sonuçları inceledik. Yapılan çalışmaların eksik yönlerine, kullanılan yöntemlerin birbirlerine karşı zayıf ve güçlü yönlerine değindik. Sonuç ve değerlendirme bölümünde hiper-parametrelerin seçiminde dikkat edilmesi gereken noktalara, günümüzde sıklıkla kullanılan yöntemlere ve ileride kullanılabilecek metodolojilere değindik.

Keywords

• Derin Öğrenme,Konvolüsyonel Sinir Ağları,Sezgisel Algoritmalar,Hiper-parametre Optimizasyonu

References

1. [1] Goodfellow, Ian, et al. Deep learning. Vol. 1. Cambridge: MIT press, 2016.
2. [2] Öztemel, Ercan. "Yapay Sinir Ağlari." PapatyaYayincilik, Istanbul (2003).
3. [3] McCulloch, Warren S., and Walter Pitts. "A logical calculus of the ideas immanent in nervous activity." The bulletin of mathematical biophysics 5.4 (1943): 115-133.
4. [4] Farley, B. W. A. C., and W. Clark. "Simulation of self-organizing systems by digital computer." Transactions of the IRE Professional Group on Information Theory 4.4 (1954): 76-84.
5. [5] ÜLKER, Erkan. "Derin Öğrenme ve Görüntü Analizinde Kullanılan Derin Öğrenme Modelleri." Gaziosmanpaşa Bilimsel Araştırma Dergisi 6.3: 85-104 (2017).
6. [6] Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." Proceedings of the thirteenth international conference on artificial intelligence and statistics. 2010.
7. [7] Karlik, Bekir, and A. Vehbi Olgac. "Performance analysis of various activation functions in generalized MLP architectures of neural networks." International Journal of Artificial Intelligence and Expert Systems 1.4 (2011): 111-122.
8. [8] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.

9. [9] He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." Proceedings of the IEEE international conference on computer vision. 2015.
10. [10] Tang, Yichuan. "Deep learning using linear support vector machines." arXiv preprint arXiv:1306.0239 (2013).
11. [11] Alpaydin, Ethem. Introduction to machine learning. MIT press, 2009.
12. [12] LeCun, Yann, et al. "Handwritten digit recognition with a back-propagation network." Advances in neural information processing systems. 1990.
13. [13] Bottou, Léon. "Large-scale machine learning with stochastic gradient descent." Proceedings of COMPSTAT'2010. Physica-Verlag HD, 2010. 177-186.
14. [14] Kingma, Diederik P., and Jimmy Lei Ba. "Adam: Amethod for stochastic optimization." Proc. 3rd Int. Conf. Learn. Representations. 2014.
15. [15] Zeiler, Matthew D. "ADADELTA: an adaptive learning rate method." arXiv preprint arXiv:1212.5701 (2012).
16. [16] Ruder, Sebastian. "An overview of gradient descent optimization algorithms." arXiv preprint arXiv:1609.04747 (2016).
17. [17] LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324.
18. [18] Ijjina, Earnest Paul, and Krishna Mohan Chalavadi. "Human action recognition using genetic algorithms and convolutional neural networks." Pattern recognition 59 (2016): 199-212.
19. [19] Dufourq, Emmanuel, and Bruce A. Bassett. "EDEN: Evolutionary deep networks for efficient machine learning." Pattern Recognition Association of South Africa and Robotics and Mechatronics (PRASA-RobMech), 2017. IEEE, 2017.
20. [20] Sun, Yanan, Bing Xue, and Mengjie Zhang. "Evolving deep convolutional neural networks for image classification." arXiv preprint arXiv:1710.10741 (2017).
21. [21] da Silva, Giovanni LF, et al. "Lung nodules diagnosis based on evolutionary convolutional neural network." Multimedia Tools and Applications 76.18 (2017): 19039-19055.
22. [22] Bochinski, Erik, Tobias Senst, and Thomas Sikora. "Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms." Image Processing (ICIP), 2017 IEEE International Conference on. IEEE, 2017.
23. [23] Fujino, Saya, Naoki Mori, and Keinosuke Matsumoto. "Deep convolutional networks for human sketches by means of the evolutionary deep learning." Fuzzy Systems Association and 9th International Conference on Soft Computing and Intelligent Systems (IFSA-SCIS), 2017 Joint 17th World Congress of International. IEEE, 2017.
24. [24] Lopez-Rincon, Alejandro, et al. "Evolutionary optimization of convolutional neural networks for cancer miRNA biomarkers classification." Applied Soft Computing 65 (2018): 91-100.
25. [25] Ma, Benteng, and Yong Xia. "Autonomous Deep Learning: A Genetic DCNN Designer for Image Classification." arXiv preprint arXiv:1807.00284 (2018).
26. [26] Assunçao, Filipe, et al. "DENSER: Deep Evolutionary Network Structured Representation." arXiv preprint arXiv:1801.01563(2018).
27. [27] Baldominos, Alejandro, Yago Saez, and Pedro Isasi. "Evolutionary convolutional neural networks: An application to handwriting recognition." Neurocomputing 283 (2018): 38-52.
28. [28] Lorenzo, Pablo Ribalta, et al. "Particle swarm optimization for hyper-parameter selection in deep neural networks." Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 2017.
29. [29] Yamasaki, Toshihiko, Takuto Honma, and Kiyoharu Aizawa. "Efficient Optimization of Convolutional Neural Networks Using Particle Swarm Optimization." Multimedia Big Data (BigMM), 2017 IEEE Third International Conference on. IEEE, 2017.
30. [30] Sun, Yanan, Bing Xue, and Mengjie Zhang. "A Particle Swarm Optimization-based Flexible Convolutional Auto-Encoder for Image Classification." arXiv preprint arXiv:1712.05042 (2017).
31. [31] Krizhevsky, Alex, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Vol. 1. No. 4. Technical report, University of Toronto, 2009.
32. [32] Coates, Adam, Andrew Ng, and Honglak Lee. "An analysis of single-layer networks in unsupervised feature learning." Proceedings of the fourteenth international conference on artificial intelligence and statistics. 2011.
33. [33] Fei-Fei, Li, Rob Fergus, and Pietro Perona. "Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories." Computer vision and Image understanding 106.1 (2007): 59-70.
34. [34] Xiao, Han, Kashif Rasul, and Roland Vollgraf. "Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms." arXiv preprint arXiv:1708.07747 (2017).
35. [35] da Silva, Giovanni Lucca França, et al. "Convolutional neural network-based PSO for lung nodule false positive reduction on CT images." Computer methods and programs in biomedicine162 (2018): 109-118.
36. [36] Nalepa, Jakub, and Pablo Ribalta Lorenzo. "Convergence Analysis of PSO for Hyper-Parameter Selection in Deep Neural Networks." International Conference on P2P, Parallel, Grid, Cloud and Internet Computing. Springer, Cham, 2018.
37. [37] Wang, Bin, et al. "Evolving Deep Convolutional Neural Networks by Variable-length Particle Swarm Optimization for Image Classification." arXiv preprint arXiv:1803.06492 (2018).
38. [38] Wang B. et al. “A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification” (2018).
39. [39] Lee, Woo-Young, Seung-Min Park, and Kwee-Bo Sim. "Optimal hyperparameter tuning of convolutional neural networks based on the parameter-setting-free harmony search algorithm." Optik172 (2018): 359-367.
40. [40] Ayumi, Vina, et al. "Optimization of convolutional neural network using microcanonical annealing algorithm." Advanced Computer Science and Information Systems (ICACSIS), 2016 International Conference on. IEEE, 2016.
41. [41] Rere, L. M., Mohamad Ivan Fanany, and Aniati Murni Arymurthy. "Metaheuristic algorithms for convolution neural network." Computational intelligence and neuroscience 2016 (2016).
42. [42] Bergstra, James, and Yoshua Bengio. "Random search for hyper-parameter optimization." Journal of Machine Learning Research 13.Feb (2012): 281-305.
43. [43] Domhan, Tobias, Jost Tobias Springenberg, and Frank Hutter. "Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves." IJCAI. Vol. 15. 2015.
44. [44] Saranyaraj, D., M. Manikandan, and S. Maheswari. "A deep convolutional neural network for the early detection of breast carcinoma with respect to hyper-parameter tuning." Multimedia Tools and Applications (2018): 1-26.
45. [45] Neary, Patrick. "Automatic Hyperparameter Tuning in Deep Convolutional Neural Networks Using Asynchronous Reinforcement Learning." 2018 IEEE International Conference on Cognitive Computing (ICCC). IEEE, 2018.
46. [46] van Stein, Bas, Hao Wang, and Thomas Bäck. "Automatic Configuration of Deep Neural Networks with EGO." arXiv preprint arXiv:1810.05526 (2018).
47. [47] Hinz, Tobias, et al. "Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks." International Journal of Computational Intelligence and Applications (2018): 1850008.
48. [48] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.
49. [49] “Caltech-101” son güncelleme 5 Nisan, 2006, http://www.vision.caltech.edu/Image_Datasets/Caltech101/.
50. [50] Larochelle, Hugo, et al. "An empirical evaluation of deep architectures on problems with many factors of variation." Proceedings of the 24th international conference on Machine learning: 473-480. ACM, 2007.
51. [51] Cohen, Gregory, et al. "EMNIST: an extension of MNIST to handwritten letters." arXiv preprint arXiv:1702.05373 (2017).
52. [52] Eitz, Mathias, James Hays, and Marc Alexa. "How do humans sketch objects?." ACM Trans. Graph. 31.4 (2012): 44-1.
53. [53] LeCun, Yann, Corinna Cortes, and C. J. Burges. "MNIST handwritten digit database." AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist 2 (2010).
54. [54] Reddy, Kishore K., and Mubarak Shah. "Recognizing 50 human action categories of web videos." Machine Vision and Applications 24.5 (2013): 971-981.
55. [55] Armato III, Samuel G., et al. "The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans." Medical physics 38.2 (2011): 915-931.
56. [56] “Cancer Genome Atlas - miRNASeq” son güncelleme 20 Kasım, 2018, http://cancergenome.nih.gov/.
57. [57] Holland, J. H. (1992). Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT press.
58. [58] Holland, J. H. (1992). Genetic algorithms. Scientific american, 267(1), 66-73.
59. [59] Goldberg, D. E., & Holland, J. H. (1988). Genetic algorithms and machine learning. Machine learning, 3(2), 95-99.
60. [60] Eberhart, R., & Kennedy, J. (1995, October). A new optimizer using particle swarm theory. In Micro Machine and Human Science, 1995. MHS'95., Proceedings of the Sixth International Symposium on (pp. 39-43). IEEE.
61. [61] Kennedy, J. (2011). Particle swarm optimization. In Encyclopedia of machine learning (pp. 760-766). Springer, Boston, MA.
62. [62] Storn, R., & Price, K. (1997). Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization, 11(4), 341-359.
63. [63] Keskintürk, T. (2006). Diferansiyel gelişim algoritması. İstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi, 5(9), 85-99.
64. [64] Goffe, William L., Gary D. Ferrier, and John Rogers. "Global optimization of statistical functions with simulated annealing." Journal of econometrics 60.1-2 (1994): 68-69.
65. [65] Geem, Z. W., Kim, J. H., & Loganathan, G. V. (2001). A new heuristic optimization algorithm: harmony search. simulation, 76(2), 60-68.
66. [66] Sinecen, M., Kaya, B., Yıldız, Ö. (2017). Artificial Neural Network Based Early Warning System For Aydin Province Towards Air Factors Which Primarily Affect Human Health. Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım ve Teknoloji, 5 (4), 121-131. DOI: 10.29109/http-gujsc-gazi-edu-tr.304938