Bayesian Learning based Gaussian Approximation for Artificial Neural Networks

Abstract

In the nonlinear systems, the pre-knowledge about the exact functional structure between inputs and outputs is mostly either unavailable or insufficient. In this case, the artificial neural networks (ANNs) are useful tools to estimate this functional structure. However, the traditional ANNs with the sum squared error suffer from the approximation and estimation errors in the high dimensional and excessive nonlinear cases. In this context, Bayesian neural networks (BNNs) provide a natural way to alleviate these issues by means of penalizing the excessive complex models. Thus, this approach allows estimating more reliable and robust models in the regression analysis, time series, pattern recognition problems etc. This paper presents a Bayesian learning approach based on Gaussian approximation which estimates the parameters and hyperparameters in the BNNs efficiently. In the application part, the proposed approach is compared with the traditional ANNs in terms of their estimation and prediction performances over an artificial data set.

Keywords

• Bayesian Neural Networks,Bayesian Learning,Gaussian Approach,Fixed Hyperparameters,Gradient based Algorithms

References

1. W. L. Buntine, A. S. Weigend, Bayesian Back-Propagation, Complex Systems 5(6) (1991), 603–643.
2. D. J. C. Mackay, A Practical Bayesian Framework for Back Propagation Networks, Neural Computation 4(3) (1992), 448–472.
3. G. E. Hinton, D. V. Camp, Keeping Neural Networks Simple by Minimizing The Description Length of The Weights, In Proceedings of the Sixth Annual Conference on Computational Learning Theory, (1993), pp. 5-13.
4. R. M. Neal, Bayesian Training of Back-Propagation Networks by the Hybrid Monte Carlo Method, Technical Report CRG-TR-92-1, Dept. of Computer Science, University of Toronto, (1992).
5. S. Duane, A. D. Kennedy, B. J. Pendleton, D. Roweth, Hybrid Monte Carlo, Physics Letters B, 195(2) (1987), 216-222.
6. D. J. C. Mackay, Probable Networks and Plausible Predictions-A Review of Practical Bayesian Methods for Supervised Neural Networks, Network: Computation in Neural Systems, 6(3) (1995), 469-505.
7. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press (reprinted 2010), 1995.
8. R. M. Neal, Bayesian Learning for Neural Networks, New York, Springer, 1996.

9. D. Rios Insua, P. Muller, Feed-forward Neural Networks for Nonparametric Regression, Technical Report 98.02., Institute of Statistics and Decision Sciences, Duke University, (1998).
10. A. D. Marrs, An Application of Reversible-Jump MCMC to Multivariate Spherical Gaussian Mixtures. Advances in Neural Information Processing Systems 10 (1998), 577-583.
11. C. C Holmes, B. K. Mallick, Bayesian Radial Basis Functions of Variable Dimension, Neural Computation. 10(5) (1998), 1217-1233.
12. P. J. Green, Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination, Biometrika, 82 (1995), 711-732.
13. S. Richardson, P. J. Green, On Bayesian Analysis of Mixtures with an Unknown Number of Components, Journal of the Royal Statistical Society B, 59(4) (1997), 731- 792.
14. J. F. G. Freitas, Bayesian Methods for Neural Networks, Phd. Thesis, Trinity College University of Cambridge and Cambridge University Engineering Department, UK, 2000.
15. F. Liang, W.H. Wong, Real-Parameter Evolutionary Monte Carlo with Applications to Bayesian Mixture Models. J. Am. Stat. Assoc. 96 (454) (2001), 653–666.
16. C. G. Chua, A. T. C. Goh, Nonlinear Modeling with Confidence Estimation using Bayesian Neural Networks, International Journal for Numerical and Analytical Methods in Geomechanics, int. J. Numer. Analy. Meth. Geomech 27 (2003), 651–667.
17. F. Liang, Bayesian Neural Networks for Nonlinear Time Series Forecasting, Statistics and Computing, 15(1), (2005), 13–29.
18. D. Lord, Y. Xie, Y. Zhang, Predicting Motor Vehicle Collisions using Bayesian Neural Network Models: An Empirical Analysis, Elsevier, Accident Analysis and Prevention, 39 (2007), 922–933.
19. J. Lampinen, A. Vehtari, Bayesian Approach for Neural Networks-Review and Case Studies, Neural Networks, 14(3) (2001), 7-24.
20. J.Vanhatalo, A. Vehtari, MCMC Methods for MLP-network and Gaussian Process and Stuff– A documentation for Matlab Toolbox MCMCstuff, Laboratory of Computational Engineering, Helsinki University of Technology, (2006).
21. T. Marwala, Bayesian Training of Neural Networks using Genetic Programming, Pattern Recognition Letters, 28 (2007), 1452-1458.
22. D.T. Mirikitani, Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling, IEEE Transactions on Neural Networks, 21 (2) (2010), 262-274.
23. M. S. Goodrich, Markov Chain Monte Carlo Bayesian Learning for Neural Networks, Selected Papers at MODSIM World 2010 Conference and Expo, NASA/CP-2011-217069/PT1 (2011), 268-290.
24. J. Martens, I. Sutskever, Learning Recurrent Neural Networks with Hessian-Free Optimization, Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA, (2011).
25. D. Niu, H. Shi and D. D. Wu. Short-term load forecasting using Bayesian neural networks learned by Hybrid Monte Carlo Algorithm, Applied Soft Computing, 12(6), (2012), 1822–1827.
26. A L. Beam, A. Motsinger-Reif and J. Doyle (2014). Bayesian Neural Networks for Genetic Association Studies of Complex Disease, arXiv:1404.3989 [q-bio.GN] , Cornell University Library.
27. O. Kocadagli, Hybrid Bayesian Neural Networks with Genetic Algorithms and Fuzzy Membership Functions, Phd. Thesis, Department of Statistics, Mimar Sinan F.A. University, Istanbul, Turkey, 2012.
28. O. Kocadagli and B. Aşıkgil. Nonlinear Time Series Forecasting with Bayesian Neural Networks. Expert Systems with Applications, 41(15), (2014), 6596-6610.
29. Kocadagli, O.. A Novel Hybrid Learning Algorithm for Full Bayesian Approach of Artificial Neural Networks, Applied Soft Computing, Elsevier, 35, (2015), 52 – 65.
30. P. Niyogi, F. Girosi, On the Relationship between Generalization Error, Hypothesis Complexity, and Sample Complexity for Radial Basis Functions, Technical Report AIM-1467, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, MA, (1994).
31. S. Geman, E. Bienenstock, R. Doursat, Neural Networks and the Bias/Variance Dilemma. Massachusetts institute of Technology, 4(1) (1992), 1-58. R. A. Jacobs, Methods for Combining Experts Probability Assessments, Neural Computation, 7(5) (1995), 867-888.
32. M. P. Perrone, Averaging/Modular Techniques for Neural Networks, In: Arbib, M. A. The Handbook of Brain Theory and Neural Networks, MIT Press, 1995. L. Wu, J. Moody, A Smoothing Regularizer for Feedforward and Recurrent Neural Networks. Neural Computation, 8(3) (1996), 461.489.
33. G. Castellano, A.M. Fanelli, M. Pelillo, An Iterative Pruning Algorithm for Feedforward Neural Networks, Neural Networks, IEEE Transactions on, 8 (1997), 519- 531.
34. G. B. Huang, P. Saratchandran, N. Sundararajan, A Generalized Growing and Pruning RBF (GGAP-RBF) Neural Network for Function Approximation. Neural Networks, IEEE Transactions on, 16 (2005), 57-67.
35. P. M. Williams, Bayesian Regularization and Pruning using A Laplace Prior, Neural Computation, 7 (1) (1995), 117-143.
36. C. M. Bishop, Pattern Recognition and Machine Learning, Springer Science + Business Media, LLC, 2006.