Multi-layer Perceptron and Pruning

Year 2017, Volume: 01 Issue: 1, 1 - 6, 22.08.2017

Cyril Voyant Christophe Paoli Marie-laure Nivet Gilles Notton Alexis Fouilloy Fabrice Motte

Abstract

A
Multi-Layer Perceptron (MLP) defines a family of artificial neural networks
often used in TS modeling and forecasting. Because of its “black box” aspect,
many researchers refuse to use it. Moreover, the optimization (often based on
the exhaustive approach where “all” configurations are tested) and learning
phases of this artificial intelligence tool (often based on the
Levenberg-Marquardt algorithm; LMA) are weaknesses of this approach (exhaustively
and local minima). These two tasks must be repeated depending on the knowledge
of each new problem studied, making the process, long, laborious and not
systematically robust. In this short communication, a pruning process is
presented. This method allows, during the training phase, to carry out an
inputs selecting method activating (or not) inter-nodes connections in order to
verify if forecasting is improved. We propose to use iteratively the popular
damped least-squares method to activate inputs and neurons. A first pass is
applied to 10% of the learning sample to determine weights significantly
different from 0 and delete other. Then a classical batch process based on LMA
is used with the new MLP. The validation is done using 25 measured meteorological
TS and cross-comparing the prediction results of the classical LMA and the
2-stage LMA.

Keywords

ANN, Forecasting, Time series, Meteorology

References

• C. Voyant, M. Muselli, C. Paoli and M.L. Nivet. "Numerical weather prediction (NWP) and hybrid ARMA/ANN model to predict global radiation", Energy, vol. 39, no. 1, (2012), pp. 341–55. doi:10.1016/j.energy.2012.01.006.
• C. Voyant, G. Notton, C. Paoli, M.L. Nivet, M. Muselli and K. Dahmani, "Numerical weather prediction or stochastic modeling: an objective criterion of choice for the global radiation forecasting", International Journal of Energy Technology and Policy, vol. 12, no. 3, (2014), pp. 01-28.
• A. Mellit, S.A. Kalogirou, L. Hontoria and S. Shaari, "Artificial intelligence techniques for sizing photovoltaic systems: A review", Renewable and Sustainable Energy Reviews, vol. 13, no. 2, (2009), pp. 406–419.
• G. Cybenko, "Approximation by superpositions of a sigmoidal function", Mathematics of Control, Signals and Systems, vol. 2, no. 4, (1989), pp.303–314.
• J. Fan and J. Pan, "A note on the Levenberg–Marquardt parameter", Applied Mathematics and Computation, vol. 207, no. 2, (2009), pp. 351-359.
• C.K. Yoo, S.W. Sung and I-B. Lee, "Generalized damped least squares algorithm", Computers & Chemical Engineering, vol. 27, no. 3, (2003), pp. 423–431.
• C. Voyant, W.W. Tamas, C. Paoli, A. Balu, M. Muselli, M.L. Nivet and G. Notton, "Time series modeling with pruned multi-layer perceptron and 2-stage damped least-squares method", 2nd International Conference on Mathematical Modeling in Physical Sciences 2013 (IC-MSQUARE 2013), Prague, Czech Republic, (2013), September 1–5.
• C. Voyant, W.W. Tamas, M.L. Nivet, G. Notton, C. Paoli, A. Balu and M. Marc, "Meteorological time series forecasting with pruned multi-layer perceptron and two-stage Levenberg-Marquardt method", International Journal of Modelling, Identification and Control, vol. 23, no. 3, (2015), pp. 287-294.
• H. Brusset, D. Depeyre, J-P. Petit and F. Haffner, "On the convergence of standard and damped least squares methods", Journal of Computational Physics, vol. 22, no. 4, (1976), pp. 534–542.
• J-P. Kreiss and E. Paparoditis, "Bootstrap methods for dependent data: A review", Journal of the Korean Statistical Society, vol. 40, no.4, (2011), pp. 357–378.
• F.M. Dias, A. Antunes, J. Vieira and A. Mota, "A sliding window solution for the on-line implementation of the Levenberg–Marquardt algorithm", Engineering Applications of Artificial Intelligence, vol. 19, no. 1, (2006), pp. 1–7.