MCDABENCH: An R Package for Benchmarking Multi-Criteria Decision Making Methods and Scenarios

Abstract

This paper introduces mcdabench, an R package for Multi-Criteria Decision Making (MCDM), which contains several useful tools, including data normalization, objective weighting, implementation of various well-known methods and their comparisons, sensitivity and stability analyses, benchmarking, aggregation, and visualization. The methods amount to 38 widely used MCDM methods from the literature. Sensitivity and stability analyses utilize measures such as the rank stability index and rank volatility. These features are enhanced by two novel contributions: Generalized gradual-weighting and random weight perturbations, both of which are important for evaluating the robustness of MCDM methods. The benchmark function applies the MCDM methods and returns a combined ranking matrix, which the comparison function then uses to calculate pairwise comparison metrics and perform statistical tests. The statistical tests include Spearman's rank correlation, Sałabun-Urbaniak similarities, and novel contributions such as Shannon’s entropy-based permutation tests and Jensen-Shannon Divergence-based bootstrap tests. The aggregation function provides a variety of methods, such as Borda Count, Kemeny-Young, Copeland, and the Markov method, to generate a final ranking. In addition, a useful "top-k" technique is used to focus on the top-k alternatives. The mcdabench package also provides built-in visualization functions to present analysis results in clean, detailed demonstrations.

Keywords

• Multi-criteria decision making
• MCDM analysis
• Benchmarking
• R package
• Industrial robots
• Robotic manipulators

Ethical Statement

Ethics committee approval was not required for this study because of there was no study on animals or humans.

References

1. Ali, A., & Meilă, M. (2012). Experiments with Kemeny ranking: What works when? Mathematical Social Sciences, 64(1), 28–40. https://doi.org/10.1016/j.mathsocsci.2011.08.008
2. Bigaret, S., Hodgett, R. E., Meyer, P., Mironova, T., & Olteanu, A. L. (2017). Supporting the multi-criteria decision aiding process: R and the MCDA package. EURO Journal on Decision Processes, 5(1–4), 169–194. https://doi.org/10.1007/s40070-017-0064-1
3. Cebeci, C. (2026). Multi-criteria evaluation with gradual-weighting and aggregation of normalised distance matrices: A case study in renewable energy grid selection. PeerJ Computer Science, 12, Article e3819. https://doi.org/10.7717/peerj-cs.3819
4. Copeland, A. H. (1951). A reasonable social welfare function. University of Michigan.
5. Demir, G., Chatterjee, P., & Pamučar, D. (2024). Sensitivity analysis in multi-criteria decision making: A state-of-the-art research perspective using bibliometric analysis. Expert Systems with Applications, 237, Article 121660. https://doi.org/10.1016/j.eswa.2023.121660
6. Gamal, A., & Mohamed, M. (2023). A hybrid MCDM approach for industrial robots selection for the automotive industry. Neutrosophic Systems with Applications, 4, 1–11.
7. Garg, R. K., & Garg, R. (2021). Decision support system for evaluation and ranking of robots using hybrid approach. IEEE Transactions on Engineering Management, 70(9), 3283–3296. https://doi.org/10.1109/TEM.2021.3079704
8. Gogodze, J. (2021). Ranking methods for multicriteria decision-making: Application to benchmarking of solvers and problems. Scientific Programming, 2021, Article 5513860. https://doi.org/10.1155/2021/5513860

9. Goswami, S. S., Behera, D. K., Afzal, A., Razak Kaladgi, A., Khan, S. A., Rajendran, P., & Asif, M. (2021). Analysis of a robot selection problem using two newly developed hybrid MCDM models of TOPSIS-ARAS and COPRAS-ARAS. Symmetry, 13(8), Article 1331. https://doi.org/10.3390/sym13081331
10. Héberger, K. (2010). Sum of ranking differences compares methods or models fairly. TrAC Trends in Analytical Chemistry, 29(1), 101–109. https://doi.org/10.1016/j.trac.2009.09.009
11. Kemeny, J. G. (1962). Mathematical models in the social sciences. MIT Press.
12. Khan, N. A., Kumar, A., & Rao, N. (2024). Hybrid decision-making program for optimal robot selection using MCDM. In 2024 International Conference on Artificial Intelligence and Emerging Technology (Global AI Summit) (pp. 173–177). IEEE. https://doi.org/10.1109/GlobalAISummit62156.2024.10947815
13. Kizielewicz, B., & Sałabun, W. (2024). The pymcdm-reidentify tool: Advanced methods for MCDA model re-identification. SoftwareX, 28, Article 101960. https://doi.org/10.1016/j.softx.2024.101960
14. Kizielewicz, B., Shekhovtsov, A., & Sałabun, W. (2023). pymcdm—The universal library for solving multi-criteria decision-making problems. SoftwareX, 22, Article 101368. https://doi.org/10.1016/j.softx.2023.101368
15. Kumar, R. (2025). A comprehensive review of MCDM methods, applications, and emerging trends. Decision Making Advances, 3(1), 185–199. https://doi.org/10.31181/dma31202569
16. Kumar, R., & Pamučar, D. (2025). A comprehensive and systematic review of multi-criteria decision-making (MCDM) methods to solve decision-making problems: Two decades from 2004 to 2024. Spectrum of Decision Making and Applications, 2(1), 178–197. https://doi.org/10.31181/sdmap21202524
17. Najafi, A., & Mirzaei, S. (2025). RMCDA: The comprehensive R library for applying multi-criteria decision analysis methods. Software Impacts, 24, Article 100762. https://doi.org/10.1016/j.simpa.2025.100762
18. Paradowski, B., Więckowski, J., & Sałabun, W. (2024). PySensMCDA: A novel tool for sensitivity analysis in multi-criteria problems. SoftwareX, 27, Article 101746. https://doi.org/10.1016/j.softx.2024.101746
19. Pereira, V., Basilio, M. P., & Santos, C. H T. (2024). Enhancing decision analysis with a large language model: pydecision, a comprehensive library of MCDA methods in Python (arXiv:2404.06370). arXiv. https://doi.org/10.48550/arXiv.2404.06370
20. Rachman, A. P., Ichwania, C., Mangkuto, R. A., Pradipta, J., Koerniawan, M. D., & Sarwono, J. (2024). Comparison of multicriteria decision-making methods for selection of optimum passive design strategy. Energy and Buildings, 314, Article 114285. https://doi.org/10.1016/j.enbuild.2024.114285
21. Rashid, T., Ali, A., & Chu, Y. M. (2021). Hybrid BW-EDAS MCDM methodology for optimal industrial robot selection. PLOS ONE, 16(2), Article e0246738. https://doi.org/10.1371/journal.pone.0246738
22. Roszkowska, E., Jefmański, B., Dudek, A., & Kusterka-Jefmańska, M. (2024). IFMCDM: An R package for intuitionistic fuzzy multi-criteria decision making methods. SoftwareX, 26, Article 101721. https://doi.org/10.1016/j.softx.2024.101721
23. Sałabun, W., & Urbaniak, K. (2020). A new coefficient of rankings similarity in decision-making problems. In V. Krzhizhanovskaya et al. (Eds.), Computational science – ICCS 2020 (Lecture Notes in Computer Science, Vol. 12138). Springer.
24. Saoud, A., Lachgar, M., Hanine, M., El Dhimni, R., El Azizi, K., & Machmoum, H. (2025). decideXpert: Collaborative system using AHP-TOPSIS and fuzzy techniques for multicriteria group decision-making. SoftwareX, 29, Article 102026. https://doi.org/10.1016/j.softx.2024.102026
25. Satman, M. H., Yıldırım, B. F., & Kuruca, E. (2021). JMcDM: A Julia package for multiple-criteria decision-making tools. Journal of Open Source Software, 6(65), Article 3430. https://doi.org/10.21105/joss.03430
26. Shanmugasundar, G., Kalita, K., Čep, R., & Chohan, J. S. (2023). Decision models for selection of industrial robots—A comprehensive comparison of multi-criteria decision making. Processes, 11(6), Article 1681. https://doi.org/10.3390/pr11061681
27. Shekhovtsov, A., Kizielewicz, B., & Sałabun, W. (2025). Version 1.3: pymcdm—The universal library for solving multi-criteria decision-making problems. SoftwareX, 29, Article 102051. https://doi.org/10.1016/j.softx.2025.102051
28. Stević, Ž., Pamučar, D., Puška, A., & Chatterjee, P. (2020). Sustainable supplier selection in healthcare industries using a new MCDM method: Measurement of alternatives and ranking according to compromise solution (MARCOS). Computers and Industrial Engineering, 140, Article 106231. https://doi.org/10.1016/j.cie.2019.106231
29. Stojčić, M., Zavadskas, E. K., Pamučar, D., Stević, Ž., & Mardani, A. (2019). Application of MCDM methods in sustainability engineering: A literature review 2008–2018. Symmetry, 11(3), Article 350. https://doi.org/10.3390/sym11030350
30. Taherdoost, H., & Madanchian, M. (2023). Multi-criteria decision making (MCDM) methods and concepts. Encyclopedia, 3(1), 77–87. https://doi.org/10.3390/encyclopedia3010006
31. Tran, N. T., Nguyen, T. S., & Tran, V. H. (2025). A hybrid triangular fuzzy MCDM model for evaluating and selecting the optimal industrial robot for manufacturing plants based on fuzzy AHP and fuzzy ARAS. Engineering, Technology and Applied Science Research, 15(4), 24270–24276. https://doi.org/10.48084/etasr.11061
32. Wilcoxon, F. (1992). Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution (pp. 196–202). Springer. https://doi.org/10.1007/978-1-4612-4380-9_16
33. Willmott, C. J. (1981). On the validation of models. Physical Geography, 2(2), 184–194. https://doi.org/10.1080/02723646.1981.10642213
34. Willmott, C. J. (1984). On the evaluation of model performance in physical geography. In Spatial statistics and models (pp. 443–460). Springer. https://doi.org/10.1007/978-94-017-3048-8_23
35. Young, P. (1995). Optimal voting rules. Journal of Economic Perspectives, 9(1), 51–64. https://doi.org/10.1257/jep.9.1.51
36. Zavadskas, E. K., & Turskis, Z. (2011). Multiple criteria decision making (MCDM) methods in economics: An overview. Technological and Economic Development of Economy, 17(2), 397–427. https://doi.org/10.3846/20294913.2011.593291