Least Squares Approach to Locally Weighted Naive Bayes Method

Year 2012, Volume: 1 Issue: 1, 71 - 80, 01.01.2012

Umut Orhan Kemal Adem Onur Comert

Abstract

This study proposes a new approach which calculates the weights of Locally Weighted Naive Bayes (LWNB) developed on Naive Bayes (NB) which is known with its simple structure. In this approach, a new equation is described by assigning a powered weight to each probabilistic factor in classic NB, and it is transformed to a linear form by using a simple assumption based on a logarithmic process, and then the weights are estimated by least squares technique. The success ratios are computed on two-class datasets from UCI database. The results show that LWNB with proposed approach is more successful than classic NB. In another analysis, it is determined that the class probability factor may sometimes damage the classification success. In addition, the effects of the attributes on the classification success are researched and according to the results the new approach is also suggested in the using as a feature selection technique of the pattern recognition problems

Keywords

Locally Weighted Naive Bayes, Least Squares, Classification, Class

References

• D. D. Lewis, Naive bayes at forty: The independence assumption in information re- trieval, In Proceedings of the tenth european conference on machine learning, Berlin, 1998, pp. 4–15.
• D. J. Hand, K. Yu, Idiot's bayes: Not so stupid after all?, International statistical review, vol. 69, no.3, 2001, pp. 385-398.
• C. H. Lee, F. Gutierrez, D. Dou, Calculating Feature Weights in Naive Bayes with Kullback-Leibler Measure, In Proceeding on Eleventh IEEE International Conference on Data Mining, 2011, pp. 1146-1151.
• S. Acid, L. M. De Campos, J. G. Castellano, Learning bayesian network classifiers: Searching in a space of partially directed acyclic graphs, Machine learning, vol. 59, no. 3, 2005, pp. 213-235. H. Alhammady, Weighted Naive Bayesian Classifier, In Proceeding on International Conference on Computer Systems and Applications, 2007, pp. 437-441.
• J. Cerquides, R. L. De Mantaras, Robust bayesian linear classifier ensembles, In Proceedings of the sixteenth european conference on machine learning, Porto, 2005, pp. 70-81.
• H. Langseth, T. D. Nielsen, Classification using hierarchical naive bayes models, Machine learning, vol. 63, no. 2, 2006, pp. 135-159.
• S. D. S. Pedro, E. R. Hruschka, N. F. F. Ebecken, WNB: A Weighted Naïve Bayesian Classifier, In Proceeding on Seventh International Conference on Intelligent Systems Design and Applications, 2007, pp. 138-142.
• G. I. Webb, J. Boughton, Z. Wang, Not so naive Bayes: Aggregating one- dependence estimators, Machine learning, vol. 58, no. 1, 2005, pp. 5-24.
• Z. Xie, W. Hsu, Z. Liu, M. L. Lee, SNNB: A selective neighborhood based naive Bayes for lazy learning, In Proceedings of the sixth pacific-asia conference on advances in knowledge discovery and data mining, Berlin, 2002, pp. 104–114.
• B. Zadrozny, C. Elkan, Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers, In Proceedings of the eighteenth international conference on machine learning, San Francisco, 2001, pp. 609-616.
• H. Zhang, L. Jiang, J. Su, Hidden naive bayes, In Proceedings of the twentieth national conference on artificial intelligence, Pittsburgh, 2005, pp. 919-924.
• F. Zheng, G. I. Webb, Finding the right family: Parent and child selection for averaged one-dependence estimators, In Proceedings of the eighteenth european conference on machine learning, Heidelberg, 2007, pp. 490-501.
• C. G. Atkeson, A. W. Moore, S. Schaal, Locally weighted learning, Artificial intelligence review, no. 11, 1997, pp. 11-73.
• E. Frank, M. Hall, B. Pfahringer, Locally weighted naive Bayes, In Proceedings of the nineteenth conference in uncertainty in artificial intelligence, Acapulco, 2003, pp. 249-256.
• H. Zhang, S. Sheng, Learning Weighted Naive Bayes with Accurate Ranking, In Proceedings of the fourth IEEE international conference on data mining, Brighton, 2004, pp. 567-570.
• B. Wang, H. Zhang, Probability based metrics for locally weighted naive bayes, In Proceedings of the twentieth Canadian conference on artificial intelligence, Montreal, 2007 pp. 180–191.
• D. R. Wilson, T. R. Martinez, Improved heterogeneous distance functions, Journal of artificial intelligence research, no. 6, 1997, pp. 1–34.
• E. Blanzieri, F. Ricci, Probability based metrics for nearest neighbor classification and case-based reasoning, In Proceedings of the third international conference on case-based reasoning research and development, Seeon Monastery, 1999, pp. 14–28.
• B. Turhan, A. Bener, Software defect prediction: Heuristics for weighted naive bayes, In Proceedings of the second international conference on software and data technologies, Barcelona, 2007, pp. 244-249.
• L. Jiang, Learning instance weighted naive bayes from labeled and unlabeled data, Journal of intelligent information systems, vol. 38, no. 1, 2012, pp. 257-268.
• C. M. Bishop, Pattern recognition and machine learning, Springer, 2006.
• C. Merz, P. Murphy, D. Aha, UCI repository of machine learning databases, Irvine: Department of ICS, University of California, accessed 10 July 2012, http://www.ics.uci.edu/mlearn/MLRepository.html.