Panelist atama problemi için bir algoritma ve karar destek sistemi: TÜBİTAK örneği

Year 2021, Volume: 36 Issue: 1, 69 - 88, 01.12.2020

Bora Kat

https://doi.org/10.17341/gazimmfd.631071

Cited By: 1

Abstract

Proje önerilerinin sağlıklı bir şekilde ve proje
konusunda uzmanlığa sahip kişiler tarafından değerlendirilmesi hem kaynakların
etkin bir şekilde kullanılması hem de bu hizmetleri yürüten kurumların
güvenilirliği açısından önem taşımaktadır. Bu çalışmada, birden fazla proje
önerisinin değerlendirildiği panellerde en uygun panelist kümesinin oluşturulması
için potansiyel panelist adaylarını dinamik bir şekilde listeleyen bir
algoritma ve bu algoritmayı kullanan bir karar destek sistemi (PaneLIST)
geliştirilmiştir. MS Excel VBA tabanlı PaneLIST, ülkemizde araştırma desteklerininin
önemli bir bölümünü sağlayan ve bunu gerçekleştirirken her yıl çok sayıda panel
düzenleyen TÜBİTAK’ın verileri kullanılarak oluşturulan test panellerinde sınanmış
ve elde edilen sonuçların öngörülen kriterlerin tamamını önemli ölçüde
sağladığı görülmüştür. Ayrıca, PaneLIST’in sunduğu çözümler, uygunluk skorları
toplamını en büyükleyen (EBSkT) ve proje önerilerinin değerlendirme seviyeleri
arasındaki sapmaları en küçükleyen (EKSp) tamsayılı programlama modelleri ile
birlikte bu iki durumu bir arada ele alan üçüncü bir modelden (EBSkT-5) elde edilen
kesin ve en iyi çözümleri ile kıyaslanmıştır. PaneLIST’in, yüksek uygunluk
skorları toplamını, projeler arasında dengeli bir dağılım gözeterek
gerçekleştirdiği ve bu yönüyle EBSkT ve EKSp’de yer alan iki hedefi bir arada
gözettiği; %5’lik bir sapma kısıtı altında en büyük skoru bulmayı amaçlayan problem
(EBSkT-5) ile hemen hemen aynı (%1’in altında yakınlık) performansı gösterdiği
belirlenmiştir.

Keywords

Akran değerlendirmesi, panel değerlendirmesi, karar destek sistemi, hakem atama problemi, tamsayılı programlama, sezgisel algortima, dinamik panelist atama

An Algorithm and a Decision Support System for the Panelist Assignment Problem: The Case of TUBITAK

Abstract

Evaluation of
project proposals in a proper manner and by the people who have expertise on
the topics of the proposals is crucial not only for efficient deployment of
resources, but also for credibility of the funding organizations. In this
study, an algorithm and a decision support system (PaneLIST) are developed to
provide a dynamic list of potential panelists from which the most appropriate
set of panelists will be selected. PaneLIST, which is based on MS Excel VBA,
has been validated by using the data of TUBITAK, primary organization
responsible for research funding and conducts the comprehensive peer review activities
for a long time. The results showed that the PaneLIST satisfies the required
criteria to a great extent. Moreover, PaneLIST’s performance was compared with
the results of the two integer programming models having the objectives of maximizing
the sum of relevance scores (EBSkT) and minimizing the total deviation among
the evalution levels of the proposals (EKSp) as well as a third model (EBSkT-5)
which couples the two. The numerical experiments showed that PaneLIST attains
high sum of relevance scores with a balanced distribution in terms of
evaluation levels of proposals, thus shows regard to objectives of both EBSkT and
EKSp at the same time; moreover, the results are so close (less than 1%) to the
results of EBSkT-5 in which sum of relevance scores is maximized under a 5%
constraint on the total deviation among the evalution levels of the proposals.

Keywords

peer review, panel evaluation, decision support system, reviewer assignment problem, integer programming, heuristic algorithm, dynamic panelist assignment

References

• Aksop, C. (2018). Akademik Makale Değerlendirmesi Kapsamında Hakem Atma Süreçlerinin İncelenmesi ve Yeni Bir Sistem Önerisi. TÜBİTAK.
• Bouajaja, S., & Dridi, N. (2017). A survey on human resource allocation problem and its applications. Operational Research, 17(2), 339–369. https://doi.org/10.1007/s12351-016-0247-8
• Ceylan, D., Saatçioǧlu, Ö., & Sepil, C. (1994). An algorithm for the committee construction problem. European Journal of Operational Research, 77(1), 60–69. https://doi.org/10.1016/0377-2217(94)90028-0
• Cook, W. D., Golany, B., Kress, M., Penn, M., & Raviv, T. (2005). Optimal allocation of proposals to reviewers to facilitate effective ranking. Management Science, 51(4), 655–661. https://doi.org/10.1287/mnsc.1040.0290
• Daş, G., & Göçken, T. (2014). A fuzzy approach for the reviewer assignment problem. Computers & Industrial Engineering, 72, 50–57. https://doi.org/10.1016/J.CIE.2014.02.014
• Dell’amico, M., & Martello, S. (1997). Linear assignment. In S. Martello, M. Dell’amico, & F. Maffioli (Eds.), Annotated bibliographies in combinatorial optimization. Chichester, England: John Wiley & Sons Ltd.
• Dumais, S., & Nielsen, J. (1992). Automating the assignment of submitted manuscripts to reviewers. Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
• ESF. (2011). European Peer Review Guide. Strasbourg.
• Fan, Z., Chen, Y., Ma, J., & Zhu, Y. (2009). Decision support for proposal grouping: A hybrid approach using knowledge rule and genetic algorithm. Expert Systems with Applications, 36(2), 1004–1013. https://doi.org/10.1016/J.ESWA.2007.11.011
• Garg, N., Kavitha, T., Kumar, A., Mehlhorn, K., & Mestre, J. (2010). Assigning Papers to Referees. Algorithmica, 58(1), 119–136. https://doi.org/10.1007/s00453-009-9386-0
• Goldsmith, J., & Sloan, R. (2007). The AI conference paper assignment problem. Proc. AAAI Workshop on Preference Handling for Artificial Intelligence. Vancouver.
• Gupta, S., & Punnen, A. (1988). Minimum deviation problems. Operations Research Letters, 7(4), 201–204. https://doi.org/10.1016/0167-6377(88)90029-6
• Hartvigsen, D., Wei, J., & Czuchlewski, R. (1999). The Conference Paper-Reviewer Assignment Problem. Decision Sciences, 30(3), 865–876. https://doi.org/10.1111/j.1540-5915.1999.tb00910.x
• Hettich, S., & Pazzani, M. (2006). Mining for proposal reviewers. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’06, 862. https://doi.org/10.1145/1150402.1150521
• Janak, S., Taylor, M., Floudas, C., Burka, M., & Mountziaris, T. (2006). Novel and Effective Integer Optimization Approach for the NSF Panel-Assignment Problem: A Multiresource and Preference-Constrained Generalized Assignment Problem. Industrial & Engineering Chemistry Research, 45(1), 258–265. https://doi.org/10.1021/ie050478h
• Jin, J., Niu, B., Ji, P., & Geng, Q. (2018). An integer linear programming model of reviewer assignment with research interest considerations. Annals of Operations Research, 1–25. https://doi.org/10.1007/s10479-018-2919-7
• Karimzadehgan, M., & Zhai, C. (2012). Integer linear programming for Constrained Multi-Aspect Committee Review Assignment. Information Processing & Management, 48(4), 725–740. https://doi.org/10.1016/J.IPM.2011.09.004
• Karimzadehgan, M., Zhai, C., & Belford, G. (2008). Multi-aspect expertise matching for review assignment. Proceeding of the 17th ACM Conference on Information and Knowledge Mining - CIKM ’08, 1113. https://doi.org/10.1145/1458082.1458230
• Kuhn, H. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1–2), 83–97. https://doi.org/10.1002/nav.3800020109
• Liu, O., Wang, J., Ma, J., & Sun, Y. (2016). An intelligent decision support approach for reviewer assignment in R&D project selection. Computers in Industry, 76, 1–10. https://doi.org/10.1016/J.COMPIND.2015.11.001
• Liu, X., Suel, T., & Memon, N. (2014). A robust model for paper reviewer assignment. Proceedings of the 8th ACM Conference on Recommender Systems - RecSys ’14, 25–32. https://doi.org/10.1145/2645710.2645749
• Ma, J., Xu, W., Sun, Y., Turban, E., Wang, S., & Liu, O. (2012). An Ontology-Based Text-Mining Method to Cluster Proposals for Research Project Selection. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 42(3), 784–790. https://doi.org/10.1109/TSMCA.2011.2172205
• Moawad, M., Maher, M., Awad, A., & Sakri S. (2019). MINARET: A Recommendation Framework for Scientific Reviewers. 22nd International Conference on Extending Database Technology (EDBT).
• Mungen, A., Gundogan, E., Alhajj, R., & Kaya, M. (2018). A Novel Local Propagation Based Expert Finding Method. 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), 1–7. https://doi.org/10.1109/IDAP.2018.8620860
• Nguyen, J., Sánchez-Hernández, G., Agell, N., Rovira, X., & Angulo, C. (2018). A decision support tool using Order Weighted Averaging for conference review assignment. Pattern Recognition Letters, 105, 114–120. https://doi.org/10.1016/J.PATREC.2017.09.020
• Pentico, D. W. (2007). Assignment problems: A golden anniversary survey. European Journal of Operational Research, 176(2), 774–793. https://doi.org/10.1016/J.EJOR.2005.09.014
• Protasiewicz, J., Pedrycz, W., Kozłowski, M., Dadas, S., Stanisławek, T., Kopacz, A., & Gałężewska, M. (2016). A recommender system of reviewers and experts in reviewing problems. Knowledge-Based Systems, 106, 164–178. https://doi.org/10.1016/J.KNOSYS.2016.05.041
• Selçuk Doğan, G. H. (2012). Expert Finding in Domains with Unclear Topics. Middle East Technical University.
• Sun, Y., Ma, J., Fan, Z., & Wang, J. (2007). A Hybrid Knowledge and Model Approach for Reviewer Assignment. 2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07). https://doi.org/10.1109/HICSS.2007.17
• Tayal, D., Saxena, P., Sharma, A., Khanna, G., & Gupta, S. (2014). New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions. Applied Intelligence, 40(1), 54–73. https://doi.org/10.1007/s10489-013-0445-5
• Üçer, S. (2011). Bilimsel Değerlendirmeler için Performans Varisi Tabanlı Bir Değerlendirme Sistemi Geliştirilmesi. TÜBİTAK.
• Wang, F., Chen, B., & Miao, Z. (2008). A Survey on Reviewer Assignment Problem. In New Frontiers in Applied Artificial Intelligence (pp. 718–727). https://doi.org/10.1007/978-3-540-69052-8_75
• Xu, Y., Ma, J., Sun, Y., Hao, G., Xu, W., & Zhao, D. (2010). A decision support approach for assigning reviewers to proposals. Expert Systems with Applications, 37(10), 6948–6956. https://doi.org/10.1016/J.ESWA.2010.03.027
• Yeşilçimen, A., & Yıldırım, E. (2019). An alternative polynomial-sized formulation and an optimization based heuristic for the reviewer assignment problem. European Journal of Operational Research, 276(2), 436–450. https://doi.org/10.1016/J.EJOR.2019.01.035
• Yıldırım, E., Aykanat, C., Oruç, A., Atmaca, A., Kayaaslan, E., & Koca, E. (2012). Geniş Kapsamlı Proje Değerlendirme ve Seçim Sistemi.
• Yunhong, X., & Xianli, Z. (2016). A LDA model based text-mining method to recommend reviewer for proposal of research project selection. 2016 13th International Conference on Service Systems and Service Management (ICSSSM), 1–5. https://doi.org/10.1109/ICSSSM.2016.7538568
• Zhao, H., Tao, W., Zou, R., & Xu, C. (2018). Construction and Application of Diversified Knowledge Model for Paper Reviewers Recommendation. https://doi.org/10.1007/978-981-13-2206-8_11