Maximum Entropy and Bayesian Approach to Constructing Probability Distributions

Abstract

Understanding the probability distribution ascribed to a given dataset enables commentary on the characteristics of the underlying population and formulation of prospective inferences. If the dataset is sufficiently large, the law of large numbers or the central limit theorem can be employed to ascertain the probability distribution of the dataset. However, in instances where the sample size is relatively limited, estimating the PDF of the population becomes difficult. Furthermore, determining the PDF that best describes the available data introduces an additional level of complexity to the analysis. Failure to consider this complexity can have significant consequences. This paper addresses this challenge by exploring the use of maximum entropy and Bayesian logical inference. The likelihood function is obtained with maximum entropy, and a priori distributions of means and standard deviations are assigned. The posterior distribution functions are constructed using Bayesian logical inference. The distributions derived theoretically were then applied to the milk yields of seven dairy cows. Subsequently, the performance of the Bayesian maximum entropy method (BME) is compared with that of the existing bootstrap method. The simulation results demonstrate the effectiveness, flexibility, and robustness of the BME method in obtaining the mean and variance distributions of the data when the number of data is small.

Keywords

• Bayesian Logical Inference
• Probability Distributions
• Priors
• Maximum Entropy
• Moment-Constrained Parametric modelling
• Bootstrap
• Optimization

References

1. Alzaatreh, A., Lee, C., & Famoye, F. (2013). A new method for generating families of continuous distributions. Metron,71(1),63-79. doi:10.1007/s40300-013-0007-y google scholar
2. Ben-Naim, A. (2008). A farewell to entropy: Statistical thermodynamics based on information. World Scientific Publishing, 9-19. ISBN-13 978-981-270-706-2. https://doi.org/10.1142/6469 google scholar
3. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. New York, NY, USA: Springer. https://cds.cern.ch/record/998831/files/9780387310732_TOC.pdf google scholar
4. Caticha, A. (2021). Entropy, information, and the updating of probabilities. Entropy 2021, 23(7). doi: 10.3390/e23070895 google scholar
5. Cevri, M., & Ustundag, D. (2012). Bayesian recovery of sinusoids from noisy data with parallel tempering. IET Signal Processing, 6(7), 673-683.doi:10.1049/iet-spr.2011.0335 google scholar
6. Cevri, M., & Ustundag, D. (2014). Performance evaluation of Gibbs sampling for Bayesian extracting sinusoids. Computational Problems in Engineering, In: Mastorakis, N. and Mladenov, V. (ed.), Springer V. ISBN: 978-3-319-03966-4.https://doi.org/10.1007/978-3-319- 03967-1_2 google scholar
7. Cevri, M., & Ustundag, D. (2016). Prediction the probabilities of transmission of genetic traits within Bayesian logical inference. Acta Physica Polonica, 130(1), 45-50. doi:10.12693/APHYSPOLA.130.45 google scholar
8. De Gregorio, J., Sánchez, D., & Toral, R. (2024). estimators for Markovian sequences: A comparative analysis, Entropy 2024, 26 (79). https://doi.org/10.3390/e26010079 google scholar

9. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aos/1176344552 google scholar
10. Erlander, S. (1981). Entropy in linear programs. Mathematical Programming: Series A and B, 21 (1), 137-151. doi: 10.1007/BF01584236 google scholar
11. Eugene, N., Lee, C., & Famoye, F. (2002). Beta-normal distribution and its applications. Commun. Stat. Theor. Methods, 31(4), 497–512. https://doi.org/10.1081/STA-120003130 google scholar
12. Furia, C. A., Feldt, R., & Torkar, R. (2019). Bayesian data analysis in empirical software engineering research. IEEE Transactions on Software Engineering, 47(9), 1786-1810. doi:10.1109/TSE.2019.2935974 google scholar
13. Gregory, P. (2005). Bayesian logical data analysis for the physical sciences. Cambridge University Press, ISBN: 0 521 841 50. https://doi.org/10.1017/CBO9780511791277 google scholar
14. Gull, S. F. (1989). Developments in maximum entropy data analysis. In J. Skilling (Ed.), Maximum Entropy and Bayesian Methods, Kluwer, 53-71. https://doi.org/10.1007/978-94-015-7860-8_4 google scholar
15. Gull, S. F., & Fielden, J. (1984). Bayesian non-parametric statistics. In Maximum Entropy and Bayesian Analysis in Applied Statistics, University of Calgary: Proceedings of the Fourth Maximum Entropy Workshop, 85–94. Cambridge University Press, 85-94. https://doi.org/10.1017/CBO9780511569678 google scholar
16. Gupta, R. D., &Kundu, D. (1999). Generalized exponential distributions. Australian & New Zealand Journal of Statistics, 41(2), 173–188. https://doi.org/10.1016/j.ajs.1999.010 google scholar
17. Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106, 620–630. DOI: https://doi.org/10.1103/PhysRev.106.620 google scholar
18. Jaynes, E. T. (2003). Probability theory: The logic of science, Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511790423 google scholar
19. Jennrich, R.I., & Sampson, P.F. (1976). Newton–Raphson and related algorithms for maximum likelihood variance component estimation. Technometrics, 18(1), 70–79. https://doi.org/10.1080/00401706.1976.10489395 google scholar
20. Jia, H., & Wang, L. (2024). Introducing entropy into organizational psychology: An entropy-based proactive control model. Behavior Science, 14(1), 54. https://doi.org/10.3390/bs14010054 google scholar
21. Johnson, N. L., Kotz, S., & Balakrishnan, N. (1994). Continuous univariate distributions. Volume 1, 2nd ed. New York, NY, USA: Wiley. google scholar
22. Jones, M. C. (2004). Families of distributions arising from distributions of order statistics. Test, 13(1), 1–43. https://doi.org/10.1016/j.test.2004.09.010 google scholar
23. Liu, C. (2023). Analysis of management entropy based on the law of enterprise lifecycle. Frontiers in Business, Economics and Management, 11 (3), 183-189. doi: 10.54097/fbem. v11i3.13213 google scholar
24. Karmeshu, J. (2003). Entropy measures, maximum entropy principle and emerging applications. New York: Springer. ISBN:978-3-540-36212-8. https://doi.org/10.1007/978-3-540-36212-8 google scholar
25. Marshall, A. W., & Olkin, I. (1997). A new method for adding a parameter to a family of distributions with application to the exponential and Weibull families. Biometrika, 84(3), 641–652. https://doi.org/10.1016/j.biometrika.1997.07.016 google scholar
26. McLachlan, G. J., & Peel, D. (2000). Finite mixture models. New York, NY, USA: Wiley. doi:10.1002/0471721182 google scholar
27. Natal, J., Ávila, I., Tsukahara, V.B, Pinheiro, M., & Maciel, C.D, (2021). Entropy: From thermodynamics to information processing. Entropy 2021(23), 1340. doi:10.3390/e23101340 google scholar
28. Olbrys, J., & Komar, N. (2023). Symbolic encoding methods with entropy-based applications to financial time series analyses. Entropy 2023, 25(7), 1009. doi:10.3390/e25071009 google scholar
29. Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., & Lakshminarayanan, B. (2021). Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57), 1–64. https://doi.org/10.1016/j.jmlr.2021.01.016 google scholar
30. Robert, H.S. (2017). Thermodynamics, statistical mechanics and entropy. Entropy 2017(19): 603. https://doi.org/10.3390/e19110603 google scholar
31. Shafii, B., & Price, J. W. (2016). Probability distributions for the mean and variance using maximum entropy and Bayesian analysis. International Journal of Applied and Experimental Mathematics, 1:103. https://doi.org/10.15344/2456-8155/2016/103 google scholar
32. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379-423. DOI: 10.1002/j.1538-7305.1948.tb01338.x google scholar
33. Trubey, P., & Sansó, B. (2024). Bayesian non-parametric inference for multivariate peaks-over-threshold models. Entropy 2024, 26(4), 335. https://doi.org/10.3390/e26040335 google scholar