Statistical Relational Learning: A State-of-the-Art Review

Abstract

The objective of this paper is to review the state-of-the-art of statistical relational learning (SRL) models developed to deal with machine learning and data mining in relational domains in presence of missing, partially observed, and/or noisy data. It starts by giving a general overview of conventional graphical models, first-order logic and inductive logic programming approaches as needed for background. The historical development of each SRL key model is critically reviewed. The study also focuses on the practical application of SRL techniques to a broad variety of areas and their limitations.

Keywords

• Statistical relational learning,probabilistic graphical models,inductive logic programming,probabilistic inductive logic programming

References

1. [1] Ben-Gal, I., “Bayesian networks”, Encyclopedia of statistics in quality and reliability (2008).
2. [2] Biba, M., “Integrating Logic and Probability: Algorithmic Improvements in Markov Logic Networks”. PhD thesis, University of Bari, Italy (2009).
3. [3] Bozcan, B., Kalkan, S., “Cosmo: Contextualized scene modeling with boltzmann machines”, Robotics and Autonomous Systems (2019) : 132–148.
4. [4] Chandra, S., Sahs, J., Khan, L., Thuraisingham, B., Aggarwal, C., “Stream mining using statistical relational learning”, In Data Mining (ICDM), IEEE International Conference on (2014), IEEE (2014) : 743–748.
5. [5] Cohen, W., Natarajan, S., “Relational restricted boltzmann machines: A probabilistic logic learning approach”, In Inductive Logic Programming: 27th International Conference, ILP 2017, Orléans, France, September 4-6, 2017, Revised Selected Papers, volume 10759, Springer (2018) : 94.
6. [6] Cussens, J., “Parameter estimation in stochastic logic programs”, Machine Learning 44(3) (2001) : 245–271.
7. [7] Dai, B., Zhang, Y., Lin, D., “Detecting visual relationships with deep relational networks”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017) : 3076–3086.
8. [8] Das, M., Dhami, D.S., Kunapuli, G., Kersting, K., Natarajan S., “Fast relational probabilistic inference and learning”, Approximate counting via hypergraphs (2019).

9. [9] Davis, J., Burnside, E. S., de Castro Dutra, I., Page, D., Ramakrishnan, R., Vitor Santos Costa, V.S., Shavlik, J.W., “View learning for statistical relational learning: With an application to mammography”. In IJCAI, Citeseer (2005) : 677–683.
10. [10] Davis, J., Ong, I.M., Struyf, J., Burnside, E.S., Page, D., Costa, V.S., “Change of representation for statistical relational learning”, In IJCAI (2007) : 2719–2726,.
11. [11] Dehbi, Y., Hadiji, F., Gröger, G., Kersting, K., Plümer, L., “Statistical relational learning of grammar rules for 3d building reconstruction”, Transactions in GIS (2016).
12. [12] De Raedt, L., Dietterich, T., Getoor, L., Muggleton, S.H., “Probabilistic, logical and relational learning-towards a synthesis”, In Dagstuhl Seminar Proceedings (2005).
13. [13] De Raedt, L., Kersting, K., “Probabilistic logic learning”, ACM SIGKDD Explorations Newsletter 5(1) (2003) : 31–48.
14. [14] De Raedt, L., Kersting, K., “Probabilistic inductive logic programming”, In International Conference on Algorithmic Learning Theory”, Springer (2004) : 19–36.
15. [15] De Raedt, L., Kersting, K., “Probabilistic inductive logic programming”, In Probabilistic Inductive Logic Programming, Springer (2008) : 1–27.
16. [16] De Raedt, L., Kersting, K., “Statistical relational learning”, In Encyclopedia of Machine Learning, Springer (2011) : 916–924.
17. [17] De Raedt, L., “Logical settings for concept-learning”, Artificial Intelligence 95(1) (1997) : 187–201.
18. [18] Domingos, P., Lowd, D., “Markov logic: An interface layer for artificial intelligence”, Synthesis Lectures on Artificial Intelligence and Machine Learning 3(1) (2009) : 1–155.
19. [19] Dong, S., Liu, D., Ouyang, R., Zhu, Y., Li, L., Li, T., Liu, J., “Second-order markov assumption based bayes classifier for networked data with heterophily”, IEEE Access (2019).
20. [20] Džeroski, S., Lavrač, N., “An introduction to inductive logic programming”, In Relational data mining, Springer (2001) : 48–73.
21. [21] Dzeroski, S., “Inductive logic programming in a nutshell”. Introduction to Statistical Relational Learning”, (2007).
22. [22] Džeroski, S., “Relational data mining. Data Mining and Knowledge Discovery”, Handbook (2010) : 887–911.
23. [23] Embar, V., Sridhar, D., Farnadi, G., Getoor, L., “Scalable structure learning for probabilistic soft logic”. arXiv preprint arXiv:1807.00973 (2018).
24. [24] Fitting, M., “First-order logic and automated theorem proving”, Springer Science & Business Media (2012).
25. [25] Friedman, N., Getoor, L., Koller, D., Pfeffer, A., “Learning probabilistic relational models”, In IJCAI volume 99 (1999) : 1300–1309.
26. [26] Genesereth R.M., Nilsson J.N., “Logical foundations of artificial. Intelligence, Morgan Kaufmann 58 (1987).
27. [27] Getoor, L., Friedman, N., Koller, D., Pfeffer, A., “Learning probabilistic relational models”, In Relational data mining, Springer (2001) : 307–335.
28. [28] Getoor, L., “Introduction to statistical relational learning”, MIT press (2007).
29. [29] Heckerman, D., Chickering M,D., Meek, C., Rounthwaite, R., Kadie, C., “Dependency networks for inference, collaborative filtering, and data visualization”, Journal of Machine Learning Research 1(Oct) (2000) : 49–75.
30. [30] Jordan I.M., “Learning in graphical models”, volume 89, Springer Science & Business Media (1998).
31. [31] Katzouris, N., Michelioudakis, E., Artikis, A., Paliouras, G., “Online learning of weighted relational rules for complex event recognition”, In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer (2018) : 396–413.
32. [32] Kazemi, S.M., Poole, D., “Bridging weighted rules and graph random walks for statistical relational models”, Frontiers in Robotics and AI 5 (2018) : 8.
33. [33] Kazemi, S.M., Poole, D., “ReINN: A deep neural model for relational learning”, In Thirty-Second AAAI Conference on Artificial Intelligence (2018).
34. [34] Kersting, K., De Raedt, L., Kramer, S., “Interpreting Bayesian logic programs”, In Proceedings of the AAAI-2000 workshop on learning statistical models from relational data (2000) : 29–35.
35. [35] Kersting, K., De Raedt, L., “Basic principles of learning Bayesian logic programs”, In Probabilistic Inductive Logic Programming Springer, Berlin, Heidelberg (2008) : 189–221.
36. [36] Koller, D., Friedman, N., Getoor, L., Taskar, B., “Graphical models in a nutshell”, Introduction to statistical relational learning (2007) : 13–55.
37. [37] Koller, D., Friedman, N., “Probabilistic graphical models: principles and techniques”, MIT press (2009).
38. [38] Koller, D., Pfeffer, A., “Probabilistic frame-based systems”, In AAAI/IAAI, (1998) : 580–587.
39. [39] Li, W., Li, L., Li, Z., Cui, M., “Statistical relational learning based automatic data cleaning”, Frontiers of Computer Science 13(1) (2019) : 215–217.
40. [40] Luperto, M., Riva, A., Amigoni, F., “Semantic classification by reasoning on the whole structure of buildings using statistical relational learning techniques”, In 2017 IEEE International Conference on Robotics and Automation (ICRA) IEEE (2017) :2562–2568.
41. [41] Muggleton, S., De Raedt, L., ”Inductive logic programming: Theory and methods”, The Journal of Logic Programming 19 (1994) : 629–679.
42. [42] Muggleton, S., “Stochastic logic programs”, Advances in inductive logic programming, 32 (1996) : 254-264.
43. [43] Muggleton, S., “Inductive logic programming”, New generation computing 8(4) (1991) : 295–318.
44. [44] Muggleton, S., “Inverse entailment and progol”, New generation computing 13(3-4) (1995) : 245–286.
45. [45] Muggleton, S., “Learning stochastic logic programs”, Electron. Trans. Artif. Intell., 4(B) (2000) : 141–153.
46. [46] Murphy, K., “A brief introduction to graphical models and bayesian networks”, (1998).
47. [47] Mutlu, E.C., Oghaz, T.A., “Review on graph feature learning and feature extraction techniques for link prediction”, arXiv preprint arXiv:1901.03425, (2019).
48. [48] Natarajan, S., Bangera, V., Khot, T., Picado, J., Wazalwar, A., Costa, V. S., Caldwell, M., “Markov logic networks for adverse drug event extraction from text”, Knowledge and information systems 51(2) (2017) : 435–457.
49. [49] Natarajan, S., Kersting, K., Ip, E., Jacobs, D. R., Carr, J., “Early prediction of coronary artery calcification levels using machine learning”, In Twenty-Fifth IAAI Conference (2013).
50. [50] Natarajan, S., Khot, T., Kersting, K., Gutmann, B., Shavlik, J., “Gradient-based boosting for statistical relational learning”, The relational dependency network case, Machine Learning 86(1) (2012) : 25–56.
51. [51] Natarajan, S., et al. “Relational learning helps in three-way classification of Alzheimer patients from structural magnetic resonance images of the brain”, International Journal of Machine Learning and Cybernetics 5(5) (2014) : 659–669.
52. [52] Neville, J., Jensen, D., “Relational dependency networks”, Journal of Machine Learning Research 8(Mar) (2007) : 653–692.
53. [53] Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E., “A review of relational machine learning for knowledge graphs”, Proceedings of the IEEE 104(1) (2015) : 11–33.
54. [54] Nishani, L., Biba, M., “Statistical relational learning for collaborative filtering a State-of-the-Art Review”, In Natural Language Processing: Concepts, Methodologies, Tools, and Applications IGI Global (2020) : 688-707.
55. [55] Poon, H., Vanderwende, L., “Joint inference for knowledge extraction from biomedical literature”, In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Association for Computational Linguistics (2010) : 813–821.
56. [56] Popescul, A., Ungar, L.H., Lawrence, S., Pennock, D.M., “Statistical relational learning for document mining”, In Third IEEE International Conference on Data Mining, IEEE (2003) : 275–282.
57. [57] Popescul, A., Ungar H. L., “Statistical relational learning for link prediction”, In IJCAI workshop on learning statistical models from relational data, volume 2003 (2003).
58. [58] Quinlan, J.R., “Learning logical definitions from relations”, Machine learning 5(3) (1990) : 239–266.
59. [59] Ravkic, I., Žnidaršič, M., Ramon, J., Davis, J., “Graph sampling with applications to estimating the number of pattern embeddings and the parameters of a statistical relational model”, Data Mining and Knowledge Discovery 32(4) (2018) : 913-948.
60. [60] Richardson, M., Domingos, P., “Markov logic networks”, Machine learning 62(1-2) (2006) : 107–136.
61. [61] Riedel, S., Chun, H.W., Takagi, T., Tsujii, J.I., “A markov logic approach to bio-molecular event extraction”, In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing: Shared Task, Association for Computational Linguistics (2009) : 41–49.
62. [62] Rios, M., Specia, L., Gelbukh, A., Mitkov, R., “Statistical relational learning to recognise textual entailment”, In International Conference on Intelligent Text Processing and Computational Linguistics, Springer, Berlin, Heidelberg (2014) : 330–339.
63. [63] Rossi, A.R., “Relational time series forecasting”, The Knowledge Engineering Review 33 (2018).
64. [64] Russell, S.J., Norvig, P., “Artificial intelligence: a modern approach”, (1995).
65. [65] Schlichtkrull, M., Kipf, T.N., Bloem, P., Van Den Berg, R., Titov, I., Welling, M., “Modeling relational data with graph convolutional networks”, In European Semantic Web Conference, Springer, Cham (2018) : 593–607.
66. [66] Shapiro Y. E., “Algorithmic program debugging”, MIT press (1983).
67. [67] Sileo, D., Van de Cruys, T., Pradel, C., Muller, P., “Improving composition of sentence embeddings through the lens of statistical relational learning”, (2018).
68. [68] Skarlatidis, A., “Event recognition under uncertainty and incomplete data”, PhD thesis, Institute of Informatics (2014).
69. [69] Speichert, S., Belle, V., “Learning probabilistic logic programs in continuous domains”, arXiv preprint arXiv:1807.05527 (2018).
70. [70] Srinivasan, A., “The aleph manual”, (2001).
71. [71] Taskar, B., Abbeel, P., Wong, M.F., Koller, D., “Relational markov networks”, Introduction to statistical relational learning (2007) : 175–200.
72. [72] Teso, S., “Statistical Relational Learning for Proteomics: Function, Interactions and Evolution”, PhD thesis, University of Trento (2013).
73. [73] Verbeke, M., Van Asch, V., Morante, R., Frasconi, P., Daelemans, W., De Raedt, L., “A statistical relational learning approach to identifying evidence based medicine categories”, In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Association for Computational Linguistics (2012) : 579–589.
74. [74] Weiss, J.C., Natarajan, S., Peissig, P.L., McCarty, C.A., Page, D., “Statistical relational learning to predict primary myocardial infarction from electronic health records”, In Twenty-Fourth IAAI Conference (2012).
75. [75] Yang, S., Korayem, M., AlJadda, K., Grainger, T., Natarajan, S., “Application of statistical relational learning to hybrid recommendation systems”, arXiv preprint arXiv:1607.01050 (2016).
76. [76] Yang, S., Korayem, M., AlJadda, K., Grainger, T., Natarajan, S., “Combining content-based and collaborative filtering for job recommendation system: A cost-sensitive statistical relational learning approach”, Knowledge-Based Systems 136 (2017) : 37–45.
77. [77] Zhang, H., Marsh, D.W.R., “Towards a model-based asset deterioration framework represented by probabilistic relational models”, (2018).