Performance Of Shannon’s Maximum Entropy Distribution Under Some Restrictions: An Application On Turkey’s Annual Temperatures

Abstract

Entropy has a very important role in Statistics. In recent studies it can be seen that entropy started to take place nearly in every brunch of science. In information theory, entropy is a measure of the uncertainty in a random variable. While there are different kinds of methods in entropy, the most common maximum entropy (MaxEnt) method maximizes the Shannon’s entropy according to the restrictions which are obtained from the random variables. MaxEnt distribution is the distribution which is obtained by this method.

The purpose of this study is to calculate the MaxEnt distribution of Turkey’s Annual temperatures for last 43 years under combinations of the restrictions 1, x, x², lnx, (lnx)², ln(1+x²) and to compare this distribution with the real probability distribution by the help of Kolmogorov-Smirnov goodness of fit test. According to the results, goodness of fit statistics accept the null hypothesis that all the entropy distributions fit with the probability distribution. The results are given in related tables and figures.

Keywords

• Shannon’s Maximum Entropy Distribution, Lagrange Multipliers, Discrete Distributions

SHANNON'UN MAKSİMUM ENTROPİ DAĞILIMININ BAZI KISITLAR ALTINDAKİ PERFORMANSI: TÜRKİYE'NİN YILLIK HAVA SICAKLIKLARI ÜZERİNE BİR UYGULAMA

Öz

İstatistik biliminde entropi oldukça önemli bir yere sahiptir. Son yıllardaki çalışmalarda entropinin neredeyse bilimin her dalında yer aldığı görülebilir. İnformasyon teorisinde, Entropi, rassal bir değişkenin belirsizliğinin bir ölçüsüdür. Entropi içerisinde farklı birçok metot olmasına rağmen, en yaygın olan Maximum Entropy (MaxEnt) metodu, rassal değişkenlerden elde edilen kısıtlara bağlı olarak Shannon’un entropisini maksimize eder. MaxEnt dağılımı ise bu metot aracılığı ile elde edilen dağılımdır.Bu çalışmanın amacı, Türkiye’nin son 43 yıllık sıcaklık değerleri için 1, x, x2, lnx, (lnx)2, ln(1+x2) kısıtlarının kombinasyonları ile MaxEnt dağılımını hesaplamak ve bu dağılımı gerçek olasılık dağılımı ile Kolmogorov-Smirnov uyum iyiliği testi yardımı ile karşılaştırmaktır. Elde edilen sonuçlara göre tüm entropi dağılımlarının gerçek olasılık dağılımı ile uyum gösterdiği şeklindeki sıfır hipotezi kabul edilmektedir. Elde edilen sonuçlar ilgili tablo ve grafiklerde verilmektedir.

Anahtar Kelimeler

• Shannon’un Maksimum Entropi Dağılımı, Lagrange Çarpanları, Kesikli Dağılımlar

References

1. Brockett P.L. (1992). Information theoretic approach to actuarial science: A unification and extension of relevant theory and applications, Society of Actuaries Transactions XLIII.
2. Brockett P.L., Charnes A., Cooper W.W., Learner D., Phillips F.Y., (1995). Information theory as a unifying statistical approach for use in marketing research, European Journal of Operational Research (84) 310-329
3. Clausius R., (1865). The mechanical theory of heat-with its applications to the steam engine and to physical properties of bodies, John van Voorst, London.
4. Çiçek H., (2013). Maksimum Entropi Yöntemi ile Türkiye’deki Coğrafi Bölgelerin Yıllık Hava Sıcaklık Değerlerinin İncelenmesi, Afyon Kocatepe Üniversitesi Fen Bilimleri Enstitüsü, Yüksek Lisans Tezi.
5. Değirmenci İ., (2011). Entropi Ölçümleri ve Maksimum Entropi İlkesi, Yüksek Lisans Tezi, Hacettepe Üniversitesi.
6. Frank J., Massey, J.R., (1951). The Kolmogorov-Smirnov Test for Goodness of Fit, Journal of the American Statistical Association, Vol. 46, No. 253 pp. 68-78.
7. Jaynes E.T., (1957). Information theory and statistical mechanics. Physics Review 106, 620–630.
8. Jaynes E. T. (1980). The minimum entropy production principle. Ann. Rev. Phys. Chem. 31, 579-601.

9. Kapur J.N., Kesevan, H.K., (1992). Entropy Optimization Principle with Applications, Academic Press.
10. Khinchine A.I., (1957). Mathematical Foundations of Information Theory, Dover Publ., New York. (New translation of Khinchine's papers "The entropy concept in probability theory" and "On the fundamental theorems of information theory" originally published in Russian in Uspekhi Matematicheskikh VII (3) (1953) and XI (1) (1956), respectively).
11. Kullback S., (1959). Information Theory and Statistics, Wiley, New York.
12. Kullback S., Leibler R.A., (1951). On information and sufficiency, Annals of Mathematical Statistics 22, 79-86.
13. Lesne A., (2011). Shannon entropy: a rigorous mathematical notion at the crossroads between probability, information theory, dynamical systems and statistical physics. Under consideration for publication in Math. Struct. in Comp. Science (Source: http://preprints.ihes.fr/2011/M/M-11-04.pdf)
14. Losee R.M., (1990). The Science of Information: Measurement and Applications. Academic Press, San Diego.
15. Parzen E., (1990a). Goodness of fit tests and entropy, Department of Statistics, Texas A & M University, Tech. Report No:103.
16. Parzen E., (1990b). Unification of statistical methods for continuous and discrete data, Department of Statistics, Texas A & M University, Tech. Report No. 105.
17. Renyi A., (1961). On measures of entropy and information, Proc. 4th Berkeley Symp. Math. Statist. Probability, 1960, University of California Press, Berkeley, CA, Vol. 1, 547-561.
18. Shannon C.E., (1948). A mathematical theory of communication, Bell System Technical Journal 27, 379-423, 623-656.
19. Shannon C.E., Weaver W., (1949). The Mathematical Theory of Communication, University of Illinois Press, Urbana, Ill.
20. Shamilov A., and Kantar Mert Y. (2005), “On a distribution minimizing maximum entropy”, Ordered Statistical Data: Approximations, Bounds and Characterizations, Izmir University of Economics.
21. Shamilov A., Kantar Mert, Y., Usta, I., (2008). Use of MinMaxEnt distributions defined on basis of MaxEnt method in wind power study, Energy Conversion & Management, 49(4), 660-677.
22. Usta İ., (2006). MaxEnt ve MinxEnt Optimizasyon Prensiplerine Bağlı Nümerik İncelemeler ve İstatistiksel Uygulamalar, Yüksek Lisans Tezi, Anadolu Üniversitesi Fen Bilimleri Enstitüsü İstatistik Anabilim Dalı.
23. Usta, İ., (2009). Moment Kısıtlarına Dayalı Genelleştirilmiş Entropi Yöntemleri, Doktora Tezi, Anadolu Üniversitesi, Fen Bilimleri Enstitüsü, İstatistik Anabilim Dalı.
24. Wu X., (2003). Calculation of maximum entropy densities with application to income distribution, Journal of Econometrics (115) 347 – 354.
25. Wu X., Stengos T., (2005). Partially Adaptive Estimation via Maximum Entropy Densities, Econometrics Journal, 8(3), 352-366.
26. Wu X., Perloff, J.M., (2007). GMM estimation of a maximum entropy distribution with interval data, Journal of Econometrics, 138(2), 532-546.