Comparison of the Performance of Machine Learning Methods with Intuitionistic Fuzzy Decision Making for Effective Fake News Detection

Abstract

In this paper, the success of machine learning models applied to two datasets consisting of fake news was examined. The performance of these methods was measured using Accuracy, Precision, Recall, and F1-Score. Evaluation metrics for all models were calculated using TF-IDF and N-gram TF-IDF for Dataset 1 and Dataset 2, respectively. In continuation of the study, a decision-making mechanism was created to measure the success of machine learning methods. The success of these models was compared by creating a decision-making mechanism using intuitionistic fuzzy sets. The PROMETHEE method was used here. In the first stage of the study, the dataset samples evaluated were expressed using classical sets, while in the second stage, the success of the models according to the metrics was expressed using intuitionistic fuzzy values. This was to minimize the uncertainty in the model success results. The results obtained in the first stage of the study were evaluated in the second stage using a decision-making mechanism. In this mechanism, machine learning models represent the alternatives, while classification metrics represent the criteria. When evaluating machine learning models, experts provide subjective opinions based on each classification metric to determine the successful model.

Keywords

• Decision making
• fake news detection
• intuitionistic fuzzy set
• machine learning.

Sahte Haber Tespiti İçin Sezgisel Bulanık Karar Verme ile Makine Öğrenmesi Yöntemlerinin Performansının Karşılaştırılması

Abstract

Bu çalışmada, sahte haberlerden oluşan iki veri kümesine uygulanan makine öğrenmesi modellerinin başarısı incelenmiştir. Bu yöntemlerin performansı Doğruluk, Kesinlik, Geri Çağırma ve F1-Puanı kullanılarak ölçülmüştür. Tüm modeller için değerlendirme metrikleri, Veri Seti 1 ve Veri Seti 2 için sırasıyla TF-IDF ve N-gram TF-IDF kullanılarak hesaplanmıştır. Çalışmanın devamında, makine öğrenmesi yöntemlerinin başarısını ölçmek için bir karar verme mekanizması oluşturulmuştur. Bu modellerin başarısı, sezgisel bulanık kümeler kullanılarak bir karar verme mekanizması oluşturularak karşılaştırılmıştır. Burada PROMETHEE yöntemi kullanılmıştır. Çalışmanın ilk aşamasında, değerlendirilen veri seti örnekleri klasik kümeler kullanılarak ifade edilmiş, ikinci aşamada ise modellerin metriklere göre başarısı sezgisel bulanık değerler kullanılarak ifade edilmiştir. Bu, model başarı sonuçlarındaki belirsizliği en aza indirmektir. Çalışmanın ilk aşamasında elde edilen sonuçlar, ikinci aşamada bir karar verme mekanizması kullanılarak değerlendirilmiştir. Bu mekanizmada, makine öğrenimi modelleri alternatifleri, sınıflandırma ölçütleri ise kriterleri temsil eder. Makine öğrenmesi modellerini değerlendirirken, uzmanlar başarılı modeli belirlemek için her sınıflandırma ölçütüne dayalı öznel görüşler sunar.

Keywords

• Karar verme
• sahte haber tespiti
• sezgisel bulanık küme
• makine öğrenmesi.

References

1. [1] Conroy, N., Rubin, V. and Chen, Y., Automatic deception detection: Methods for finding fake news, Proceedings of the Association for Information Science and Technology, 2015, 52(1), pp.1-4.
2. [2] Granik, M., and Mesyura, V., Fake news detection using naive Bayes classifier, In 2017 IEEE first Ukraine conference on electrical and computer engineering (UKRCON), 2017, pp. 900-903.
3. [3] WHO, 2022. https://www.who.int/about/accountability/results/who-results-report-2020-2021
4. [4] Ahmed, H., Traore, I., and Saad, S., Detection of online fake news using n-gram analysis and machine learning techniques. In International conference on intelligent, secure, and dependable systems in distributed and cloud environments, 2017, pp. 127-138). Cham: Springer International Publishing.
5. [5] Wang, W. Y., " Liar, liar pants on fire": A new benchmark dataset for fake news detection, arXiv preprint arXiv:1705.00648, 2017.
6. [6] Khan, W., Daud, A., Khan, K., Nasir, J. A., Basheri, M., Aljohani, N., and Alotaibi, F. S., Part of speech tagging in urdu: Comparison of machine and deep learning approaches. IEEE Access, 2019, 7, pp.38918-38936.
7. [7] Hu, B., Mao, Z., and Zhang, Y., An overview of fake news detection: From a new perspective, Fundamental Research, 2025, 5(1), pp.332-346.
8. [8] Ahmad, I., Yousaf, M., Yousaf, S. and Ahmad, M. O., Fake news detection using machine learning ensemble methods, Complexity, 2020, 1, 8885861.

9. [9] Baarir, N. F., and Djeffal, A., Fake news detection using machine learning, In 2020 2nd International workshop on human-centric smart environments for health and well-being (IHSH), 2021, pp. 125-130.
10. [10] Manzoor, S. I. and Singla, J., Fake news detection using machine learning approaches: A systematic review. In 2019 3rd international conference on trends in electronics and informatics (ICOEI), 2019, pp. 230-234.
11. [11] Sharma, U., Saran, S. and Patil, S. M., Fake news detection using machine learning algorithms. International Journal of creative research thoughts (IJCRT), 2020, 8(6), pp.509-518.
12. [12] Alghamdi, J., Lin, Y. and Luo, S., A comparative study of machine learning and deep learning techniques for fake news detection, Information, 2022, 13(12), 576.
13. [13] Alghamdi, J., Luo, S. and Lin, Y., A comprehensive survey on machine learning approaches for fake news detection, Multimedia Tools and Applications, 2024, 83(17), pp.51009-51067.
14. [14] Yakkundi, S., Patil, R., Sangani, S., Goudar, R. H., Goudar, S. I. and Qazi, A., Exploring machine learning for fake news detection: techniques, tools, challenges, and future research directions, Discover Applied Sciences, 2025, 7(8), 873.
15. [15] Can, Ü. and Alatas, B., A new direction in social network analysis: Online social network analysis problems and applications, Physica A: Statistical Mechanics and its Applications, 2019, 535, 122372.
16. [16] Can, Ü. and Alataş, B., Review of Sentiment Analysis and Opinion Mining Algorithms, International Journal of Pure and Applied Sciences, 2017, 3(1), 75-111.
17. [17] Akinsola, J. E. T., Awodele, O., Kuyoro, S. O., and Kasali, F. A., Performance evaluation of supervised machine learning algorithms using multicriteria decision making techniques, In Proceedings of the International Conference on Information Technology in Education and Development (ITED), 2019, pp. 17-34.
18. [18] Tuğrul, F., Examination of Recycled Papers Using an Intuitionistic Fuzzy Set-Based MCDM. In Advanced Mathematics for the Modeling and Solution of Challenging Problems in Engineering, 2025, pp 165-178. Cham: Springer Nature Switzerland.
19. [19] Tuğrul, F., An innovative application on supermarket selection through using intuitionistic fuzzy TOPSIS method, Sakarya University Journal of Science, 2022, 26(5), pp.1029-1039.
20. [20] URL-1: https://www.kaggle.com/code/abaghyangor/fake-news#Introduction
21. [21] URL-2: https://www.kaggle.com/datasets/saurabhshahane/fake-news-classification/data
22. [22] Atanassov, K.T., Intuitionistic fuzzy sets, Fuzzy Sets and Systems, 1986, 20(1), 87-96.
23. [23] Zadeh, L.A., Fuzzy Sets. Information and Control, 1965, 8, pp.338-353.
24. [24] Brans, J.P., Lingenierie de la decision lelaboration dinstruments daidea la decision la methode PROMETHEE. In: Laidea la decision Nature, Instruments et Perspectives d Avenir (Edt. Nadeau, R., Landry, M), Les Presses de l’universite Laval, ISBN:2-7637-7084-3, Quebec, Canada.1982, pp.183-213.
25. [25] Atanassov, K.T., Intuitionistic fuzzy sets, International Journal Bioautomation, 2016, 20(S1), pp.1-6.
26. [26] Brans, J.P., Mareschal, B., and Vincke, P., PROMETHEE: a new family of outranking methods in multicriteria analysis. Operational Research, 1984, 84, pp.477-490.
27. [27] Xu, Z.S., and Yager, R.R., Some geometric aggregation operators based on intuitionistic fuzzy set, International Journal of General Systems, 2006, 35, 417-433.
28. [28] Xu, Z.S., Intuitionistic fuzzy aggregation operators, IEEE Transactions on Fuzzy Systems, 2007, 15, 1179-1187.
29. [29] Xu, Z.S., Intuitionistic preference relations and their application in group decision making, Information Sciences, 2007, 177(11), 2363-2379.
30. [30] Szmidt, E., and Kacprzyk, J., Amount of information and its reliability in the ranking of Atanassov’s intuitionistic fuzzy alternatives. In: Recent advances in decision making (Edt. Rakus-Andersson, E., Yager, R.R., Ichalkaranje, N., Jain, L) Springer, ISBN: 978-3-642-02187-9, Berlin, Heidelberg, 2009, pp.7-19.
31. [31] Vincke, P., and Brans, J.P., A preference ranking organization method. The PROMETHEE method for MCDM. Management Science, 1985, 31(6), pp.647-656.