Entropi ve Kümeleme Tabanlı Yeni Bir Hakem Değerlendirme Süreci ve Karar Destek Sistemi Önerisi

Abstract

Araştırma ve geliştirme (Ar-Ge) projelerinin değerlendirme süreçlerinin karmaşıklığı ve eksiklikleri, projelerin değerlendirilmesinde tutarlılık ve izlenebilirlik sorunlarına yol açmaktadır. Bu çalışma, Türkiye'deki Ar-Ge projelerinin değerlendirme süreçlerinde karşılaşılan temel sorunları ele alarak, nesnel, tutarlı ve uygulanabilirliği yüksek bir karar destek sistemi geliştirmeyi amaçlamaktadır. Bu bağlamda, projelerin değerlendirilmesinde kullanılan kriterler için literatür detaylı bir şekilde incelenmiş, uzmanların görüşleri alınmış, entropi yöntemi ile önem derecelerine göre sıralanmış ve değerlendirmede daha etkili olduğu görülen kriterler belirlenmiştir. Böylece, kriter sayısı 247'den 96'ya indirilmiş ve değerlendiriciye kolaylık sağlanarak daha etkili bir değerlendirme seti oluşturulmuştur. Geliştirilen karar destek sistemi, projelerin kapsamlı bir analizine olanak tanımakta ve hakemlerden yürütme kurulu üyelerine kadar tüm paydaşların daha şeffaf bir platform üzerinde değerlendirme yapmasını sağlamaktadır. Karar destek sistemi ile elde edilen hakem puanlarına bağlı olarak projeler, K-ortalama kümeleme algoritması ile dört ana gruba ayrılmıştır. Bu platform, değerlendirme süreçlerinin hızlandırılmasını ve karar vericilerin iş yükünün %31 ila %39 oranında azaltılmasını sağlamıştır. Elde edilen sonuçlar, Ar-Ge projelerinin yenilikçi yönleri, ekonomik ve ulusal katkıları, proje planları ve teknik altyapıları gibi boyutlarda daha kapsamlı ve objektif bir değerlendirme yapılmasını mümkün kılmıştır. Yeni sistem, mevcut değerlendirme süreçlerinin eksikliklerini gidererek daha yapılandırılmış ve bütüncül bir yaklaşım sunmaktadır. Geliştirilen bu sistem, gelecekte farklı Ar-Ge programlarına entegre edilerek ülkenin Ar-Ge potansiyelini ve inovasyon kapasitesini en üst seviyeye çıkarmayı hedeflemektedir.

Keywords

• Ar-Ge projeleri değerlendirme süreci
• Çok kriterli karar verme
• Entropi
• K-ortalama kümeleme
• Karar destek sistemi

A Novel Reviewer Evaluation Process and Decision Support System Proposal Based on Entropy and Clustering

Abstract

The complexity and shortcomings of the evaluation processes of research and development (R&D) projects lead to problems of consistency and traceability in the evaluation of the projects. This study aims to develop an objective, consistent, and highly applicable decision support system by addressing the main challenges encountered in the evaluation processes of R&D projects in Türkiye. In this context, the literature on evaluation criteria was thoroughly reviewed, expert opinions were collected, the criteria were ranked according to their importance using the entropy method, and the most effective criteria for evaluation were identified. Consequently, the number of criteria was reduced from 247 to 96, thereby facilitating the evaluator’s task and forming a more effective evaluation set. The developed decision support system enables comprehensive analysis of projects and allows all stakeholders—from reviewers to executive board members—to conduct evaluations on a more transparent platform. Based on the reviewer scores obtained through the system, projects were classified into four main groups using the k-means clustering algorithm. This platform has accelerated evaluation processes and reduced the workload of decision-makers by 31% to 39%. The results demonstrate that more comprehensive and objective evaluations can be made in dimensions such as the innovative aspects of R&D projects, their economic and national contributions, project planning, and technical infrastructure. The newly developed system addresses the shortcomings of existing evaluation procedures and offers a more structured and holistic approach. It is envisioned that the system can be integrated into various R&D programs in the future to maximize the country’s R&D potential and innovation capacity.

Keywords

• R&D project evaluation process
• Multi-criteria decision making
• Entropy
• K-means clustering
• Decision support system

References

1. Archer NP, Ghasemzadeh F. 1999. An integrated framework for project portfolio selection, Int J Proj Manag, 17(4): pp: 207-216.
2. Arthur D, Vassilvitskii S. 2006. k-means++: The advantages of careful seeding, Stanford, pp: 56-59.
3. Bard JF, Balachandra R, Kaufmann PE. 1988. An interactive approach to R&D project selection and termination, IEEE Trans Eng Manag, 35(3): pp: 139-146.
4. Beaujon GJ, Marin SP, McDonald GC. 2001. Balancing and optimizing a portfolio of R&D projects, Nav Res Logist, 48(1): pp: 18-40.
5. Bhattacharyya R, Kumar P, Kar S. 2011. Fuzzy R&D portfolio selection of interdependent projects, Comput Math Appl, 62(10): pp: 3857-3870.
6. Bhattacharyya R. 2015. A grey theory based multiple attribute approach for R&D project portfolio selection, Fuzzy Inf Eng, 7(2): pp: 211-225.
7. Bitman WR, Sharif N. 2008. A conceptual framework for ranking R&D projects, IEEE Trans Eng Manag, 55(2): pp: 267-278.
8. Bornmann L. 2011. Scientific peer review, Annu Rev Inf Sci Technol, 45(1): pp: 197-245.

9. Cebeci Z, Cebeci C. 2020. A fast algorithm to initialize cluster centroids in fuzzy clustering applications, Information, 11(9): pp: 446.
10. Cebeci Z, Yildiz F. 2015. Comparison of k-means and fuzzy c-means algorithms on different cluster structures, J Agric Inform, 6(3): pp: 15-59.
11. Charnes A, Cooper WW, Rhodes E. 1978. Measuring the efficiency of decision making units, Eur J Oper Res, 2(6): pp: 429-444.
12. Cheng CH, Liou JJ, Chiu CY. 2017. A consistent fuzzy preference relations based ANP model for R&D project selection, Sustainability, 9(8): pp: 1352.
13. Conka T, Vayvay O, Sennaroglu B. 2008. A combined decision model for R&D project portfolio selection, Int J Bus Innov Res, 2(2): pp: 190-202.
14. Criscuolo P, Dahlander L, Grohsjean T, Salter A. 2017. Evaluating novelty: The role of panels in the selection of R&D projects, Acad Manag J, 60(2): pp: 433-460.
15. Czajkowski AF, Jones S. 1986. Selecting interrelated R&D projects in space technology planning, IEEE Trans Eng Manag, (1): pp: 17-24.
16. de Souza DGB, dos Santos EA, Soma NY, da Silva CES. 2021. MCDM-based R&D project selection: A systematic literature review, Sustainability, 13(21): pp: 11626.
17. Demšar J, Zupan B, Leban G, Curk T. 2004. Orange: From experimental machine learning to interactive data mining, In: Knowledge Discovery in Databases: PKDD 2004, Pisa, Italy, 20 September 2004 - 24 September 2004, Springer Berlin Heidelberg, pp: 537-539.
18. Eclipse Foundation. 2024. Eclipse IDE (Version [2023 12]), URL: https://www.eclipse.org/ide/ (accessed date: June 28, 2024).
19. Eilat H, Golany B, Shtub A. 2008. R&D project evaluation: An integrated DEA and balanced scorecard approach, Omega, 36(5): pp: 895-912.
20. Forgy EW. 1965. Cluster analysis of multivariate data: efficiency versus interpretability of classifications, Biometrics, 21: pp: 768-769.
21. Ghazal TM. 2021. Performances of k-means clustering algorithm with different distance metrics, Intell Autom Soft Comput, 30(2): pp: 735-742.
22. Ghasemzadeh F, Archer NP. 2000. Project portfolio selection through decision support, Decis Support Syst, 29(1): pp: 73-88.
23. Ghorbani HR, Jolai F, Tavakkoli-Moghaddam R, Rabbani M. 2006. A comprehensive model for R&D project portfolio selection with zero-one linear goal-programming (research note), Int J Eng, 19(1): pp: 55-66.
24. Gustafsson J, Salo A. 2005. Contingent portfolio programming for the management of risky projects, Oper Res, 53(6): pp: 946-956.
25. Hassanzadeh F, Nemati H, Sun M. 2014. Robust optimization for interactive multiobjective programming with imprecise information applied to R&D project portfolio selection, Eur J Oper Res, 238(1): pp: 41-53.
26. Henig MI, Katz H. 1996. R&D project selection: A decision process approach, J Multi-Criteria Decis Anal, 5(3): pp: 169-177.
27. Henriksen AD, Traynor AJ. 1999. A practical R&D project-selection scoring tool, IEEE Trans Eng Manag, 46(2): pp: 158-170.
28. Hesarsorkh AH, Ashayeri J, Naeini AB. 2021. Pharmaceutical R&D project portfolio selection and scheduling under uncertainty: A robust possibilistic optimization approach, Comput Ind Eng, 155: pp: 107114.
29. Heyard R, Ott M, Salanti G, Egger M. 2022. Rethinking the funding line at the Swiss national science foundation: Bayesian ranking and lottery, Stat Public Policy, 9(1): pp: 110-121.
30. Heydari T, Seyed Hosseini SM, Makui A. 2016. Developing and solving a one-zero non-linear goal programming model to R&D portfolio project selection with interactions between projects, Int Bus Manag, 10(19): pp: 4516-4521.
31. Hsu YG, Tzeng GH, Shyu JZ. 2003. Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects, R&D Manag, 33(5): pp: 539-551.
32. Hünermund P, Czarnitzki D. 2019. Estimating the causal effect of R&D subsidies in a pan-European program, Res Policy, 48(1): pp: 115-124.
33. Isied M, Daneshvar S. 2024. A second chance for failed projects using data envelopment analysis based on project attractiveness factors, Mathematics, 12(23): pp: 10-15.
34. Imoto S, Yabuuchi Y, Watada J. 2008. Fuzzy regression model of R&D project evaluation, Appl Soft Comput, 8(3): pp: 1266-1273.
35. Jeng DJF, Huang KH. 2015. Strategic project portfolio selection for national research institutes, J Bus Res, 68(11): pp: 2305-2311.
36. Kahraman C, Çağrı Tolga A. 2008. Fuzzy multiattribute evaluation of R&D projects using a real options valuation model, Int J Intell Syst, 23(11): pp: 1153-1176.
37. Kaplan RS, Norton DP. 1992. The Balanced Scorecard: measures that drive performance, Harvard Business Review, January-February: pp: 71-79.
38. Karami A, Johansson R. 2013. Utilization of multi attribute decision making techniques to integrate automatic and manual ranking of options, J Inf Sci Eng, 30(2): pp: 519-534.
39. Karasakal E, Aker P. 2017. A multicriteria sorting approach based on data envelopment analysis for R&D project selection problem, Omega, 73: pp: 79-92.
40. Karaveg C, Thawesaengskulthai N, Chandrachai A. 2015. A combined technique using SEM and TOPSIS for the commercialization capability of R&D project evaluation, Decis Sci Lett, 4(3): pp: 379-396.
41. Ketchen DJ, Shook CL. 1996. The application of cluster analysis in strategic management research: an analysis and critique, Strateg Manag J, 17(6): pp: 441-458.
42. Kumar SS. 2004. AHP-based formal system for R&D project evaluation, Int J Proj Manag, 22(5): pp: 423-432.
43. Liberatore MJ. 1986. R&D project selection, Telemat Inform, 3(4): pp: 289-300.
44. Liberatore MJ. 1987. An extension of the analytic hierarchy process for industrial R&D project selection and resource allocation, IEEE Trans Eng Manag, (1): pp: 12-18.
45. Liberatore MJ. 1988. An expert support system for R&D project selection, Math Comput Model, 11: pp: 260-265.
46. Liberatore MJ, Titus GJ. 1983. The practice of management science in R&D project management, Manag Sci, 29(8): pp: 962-974.
47. Liu M, Yang WQS. 2023. Analysis of the advantages and disadvantages of four comprehensive evaluation methods, Front Bus Econ Manag, 9: pp: 162-167.
48. Madey GR, Dean BV. 1985. Strategic planning for investment in R&D using decision analysis and mathematical programming, IEEE Trans Eng Manag, (2): pp: 84-90.
49. MacQueen J. 1967. Some methods for classification and analysis of multivariate observations, In: Proc of the Fifth Berkeley Symp on Math Statist and Probab, 1(14): pp: 281-297.
50. Mardani A, Jusoh A, Zavadskas EK, Khalifah Z, Nor KM. 2015. Application of multiple-criteria decision-making techniques and approaches to evaluating of service quality: a systematic review of the literature, J Bus Econ Manag, 16(5): pp: 1034-1068.
51. Mavrotas G, Makryvelios E. 2021. Combining multiple criteria analysis, mathematical programming and Monte Carlo simulation to tackle uncertainty in R&D project portfolio selection: a case study from Greece, Eur J Oper Res, 291(2): pp: 794-806.
52. Mavrotas G, Makryvelios E. 2023. R&D project portfolio selection using the Iterative Trichotomic Approach in order to study how subjectivity of the weights is reflected in the selected projects of the final portfolio, Oper Res, 23(3): pp: 50-56.
53. Meade LM, Presley A. 2002. R&D project selection using the analytic network process, IEEE Trans Eng Manag, 49(1): pp: 59-66.
54. Miyajima S, Isshiki T, Kunugi M, Uesaka S. 2022. Can we predict successful market introduction using on-going R&D evaluation data?, fteval J Res Technol Policy Eval, (53): pp: 153-159.
55. Mohaghar A, Fathi MR, Faghih A, Turkayesh MM. 2012. An integrated approach of Fuzzy ANP and Fuzzy TOPSIS for R&D project selection: a case study, Aust J Basic Appl Sci, 6(2): pp: 66-75.
56. Mohanty RP, Agarwal R, Choudhury AK, Tiwari MK. 2005. A fuzzy ANP-based approach to R&D project selection: a case study, Int J Prod Res, 43(24): pp: 5199-5216.
57. Montajabiha M, Arshadi Khamseh A, Afshar-Nadjafi B. 2017. A robust algorithm for project portfolio selection problem using real options valuation, Int J Manag Proj Bus, 10(2): pp: 386-403.
58. Oral M, Kettani O, Lang P. 1991. A methodology for collective evaluation and selection of industrial R&D projects, Manag Sci, 37(7): pp: 871-885.
59. Oral M. 2012. Action research contextualizes DEA in a multi-organizational decision-making process, Expert Syst Appl, 39(7): pp: 6503-6513.
60. Ömürbek N, Aksoy E. 2016. Bir petrol şirketinin çok kriterli karar verme teknikleri ile performans değerlendirmesi, Süleyman Demirel Üniv İİBF Derg, 21(3): pp: 25-27.
61. Ramalho FD, Silva IS, Ekel PY, da Silva Martins CAP, Bernardes P, Libório MP. 2021. Multimethod to prioritize projects evaluated in different formats, MethodsX, 8: pp: 101371.
62. Rousseeuw PJ. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J Comput Appl Math, 20: pp: 53-65.
63. Salehi F, Al-e SMJM, Husseini SMM. 2022. A 2-phase interdependent methodology for sustainable project portfolio planning in the pharmaceutical industry, Comput Ind Eng, 174: pp: 108794.
64. Shahapure KR, Nicholas C. 2020. Cluster quality analysis using silhouette score, In: 2020 IEEE 7th Int Conf Data Sci Adv Anal (DSAA), pp: 747-748.
65. Shannon CE. 1948. A mathematical theory of communication, Bell Syst Tech J, 27(3): pp: 379-423.
66. Siksnelyte-Butkiene I, Zavadskas EK, Streimikiene D. 2020. Multi-criteria decision-making (MCDM) for the assessment of renewable energy technologies in a household: a review, Energies, 13(5): pp: 1164.
67. Stelmakh I, Shah N, Singh A. 2019. On testing for biases in peer review, Adv Neural Inf Process Syst, 32: pp: 42-56.
68. Stewart TJ. 1991. A multi-criteria decision support system for R&D project selection, J Oper Res Soc, 42(1): pp: 17-26.
69. Sun M, Barry Danfa J, Teplitskiy M. 2022. Does double‐blind peer review reduce bias? Evidence from a top computer science conference, J Assoc Inf Sci Technol, 73(6): pp: 811-819.
70. Suvvari SK. 2023. Project portfolio management: best practices for strategic alignment, Int J Prof Bus Rev, 8(12): pp: 1.
71. Syakur MA, Khotimah BK, Rochman EMS, Satoto BD. 2018. Integration k-means clustering method and elbow method for identification of the best customer profile cluster, In: IOP Conf Ser Mater Sci Eng, 336: ID: 012017.
72. Taylor BW III, Moore LJ, Clayton ER. 1982. R&D project selection and manpower allocation with integer nonlinear goal programming, Manag Sci, 28(10): pp: 1149-1158.
73. TÜBİTAK. 2023. Türkiye Bilimsel ve Teknolojik Araştırma Kurumu (TÜBİTAK) Faaliyet Raporları, URL: https://tubitak.gov.tr/sites/default/files/26720/tubitak_2023_yili_faaliyet_raporu.pdf (published date: March 29, 2024, accessed date: November 16, 2024).
74. Wang K, Wang CK, Hu C. 2005. Analytic hierarchy process with fuzzy scoring in evaluating multidisciplinary R&D projects in China, IEEE Trans Eng Manag, 52(1): pp: 119-129.
75. Wang TC, Lee HD. 2009. Developing a fuzzy TOPSIS approach based on subjective weights and objective weights, Expert Syst Appl, 36(5): pp: 8980-8985.
76. Wang Z, Bai S, Zhang X. 2024. A robust optimization model for R&D project portfolio risk response decision-making considering risk interaction and regret perception, J Ind Manag Optim, 20(12): pp: 3843-3867.
77. Wu J, Liang L, Yang F, Yan H. 2009. Bargaining game model in the evaluation of decision making units, Expert Syst Appl, 36(3): pp: 4357-4362.
78. Xu R, Wunsch D. 2008. Clustering, John Wiley & Sons, Hoboken, NJ, USA, pp: 25-30.
79. Zhang H, Gu CL, Gu LW, Zhang Y. 2011. The evaluation of tourism destination competitiveness by TOPSIS & information entropy–a case in the Yangtze River Delta of China, Tour Manag, 32(2): pp: 443-451.