Multi-layer Perceptron and Pruning

Abstract

A Multi-Layer Perceptron (MLP) defines a family of artificial neural networks often used in TS modeling and forecasting. Because of its “black box” aspect, many researchers refuse to use it. Moreover, the optimization (often based on the exhaustive approach where “all” configurations are tested) and learning phases of this artificial intelligence tool (often based on the Levenberg-Marquardt algorithm; LMA) are weaknesses of this approach (exhaustively and local minima). These two tasks must be repeated depending on the knowledge of each new problem studied, making the process, long, laborious and not systematically robust. In this short communication, a pruning process is presented. This method allows, during the training phase, to carry out an inputs selecting method activating (or not) inter-nodes connections in order to verify if forecasting is improved. We propose to use iteratively the popular damped least-squares method to activate inputs and neurons. A first pass is applied to 10% of the learning sample to determine weights significantly different from 0 and delete other. Then a classical batch process based on LMA is used with the new MLP. The validation is done using 25 measured meteorological TS and cross-comparing the prediction results of the classical LMA and the 2-stage LMA.

Keywords

• ANN,Forecasting,Time series,Meteorology

References

1. C. Voyant, M. Muselli, C. Paoli and M.L. Nivet. "Numerical weather prediction (NWP) and hybrid ARMA/ANN model to predict global radiation", Energy, vol. 39, no. 1, (2012), pp. 341–55. doi:10.1016/j.energy.2012.01.006.
2. C. Voyant, G. Notton, C. Paoli, M.L. Nivet, M. Muselli and K. Dahmani, "Numerical weather prediction or stochastic modeling: an objective criterion of choice for the global radiation forecasting", International Journal of Energy Technology and Policy, vol. 12, no. 3, (2014), pp. 01-28.
3. A. Mellit, S.A. Kalogirou, L. Hontoria and S. Shaari, "Artificial intelligence techniques for sizing photovoltaic systems: A review", Renewable and Sustainable Energy Reviews, vol. 13, no. 2, (2009), pp. 406–419.
4. G. Cybenko, "Approximation by superpositions of a sigmoidal function", Mathematics of Control, Signals and Systems, vol. 2, no. 4, (1989), pp.303–314.
5. J. Fan and J. Pan, "A note on the Levenberg–Marquardt parameter", Applied Mathematics and Computation, vol. 207, no. 2, (2009), pp. 351-359.
6. C.K. Yoo, S.W. Sung and I-B. Lee, "Generalized damped least squares algorithm", Computers & Chemical Engineering, vol. 27, no. 3, (2003), pp. 423–431.
7. C. Voyant, W.W. Tamas, C. Paoli, A. Balu, M. Muselli, M.L. Nivet and G. Notton, "Time series modeling with pruned multi-layer perceptron and 2-stage damped least-squares method", 2nd International Conference on Mathematical Modeling in Physical Sciences 2013 (IC-MSQUARE 2013), Prague, Czech Republic, (2013), September 1–5.
8. C. Voyant, W.W. Tamas, M.L. Nivet, G. Notton, C. Paoli, A. Balu and M. Marc, "Meteorological time series forecasting with pruned multi-layer perceptron and two-stage Levenberg-Marquardt method", International Journal of Modelling, Identification and Control, vol. 23, no. 3, (2015), pp. 287-294.

9. H. Brusset, D. Depeyre, J-P. Petit and F. Haffner, "On the convergence of standard and damped least squares methods", Journal of Computational Physics, vol. 22, no. 4, (1976), pp. 534–542.
10. J-P. Kreiss and E. Paparoditis, "Bootstrap methods for dependent data: A review", Journal of the Korean Statistical Society, vol. 40, no.4, (2011), pp. 357–378.
11. F.M. Dias, A. Antunes, J. Vieira and A. Mota, "A sliding window solution for the on-line implementation of the Levenberg–Marquardt algorithm", Engineering Applications of Artificial Intelligence, vol. 19, no. 1, (2006), pp. 1–7.