Tools and Methods Used in Performance Measurement in Healthcare

Abstract

In recent years, the concept of independent and impartial performance measurement, which has been developing rapidly in line with the increasing expectations and needs of patients in health systems, is of great importance in terms of creating auditable, patient-oriented, highly accountable, efficient, durable, sustainable, and accessible systems. One of the main actors of the performance evaluation process is the presentation of a structure where certain performance indicators can be defined and can be used regularly in the required areas in the health system, as well as, in order to create a holistic view of performance to clarify this process, researchers who focus on the subject continue to work on various building blocks of the health system by using different tools and methods. Although performance evaluation in health services is a difficult and complex process, it is of critical importance for improving and developing the system. In this review study, the tools and methods used in various performance evaluation studies carried out in the field of health between the years 2011-2021 were examined and summarized, and the prominent concepts in the studies were classified. As a result of the study, it was aimed to contribute to the existing literature by observing the deficiencies in the literature.

Keywords

• Healthcare systems
• performance
• performance measurement
• BSC
• DEA
• MCDM.

Sağlık Hizmetlerinde Performans Değerlendirmede Kullanılan Araç ve Yöntemler

Abstract

Son yıllarda, sağlık sistemlerinde, hastaların artan beklenti ve ihtiyaçları doğrultusunda hızla gelişmekte olan bağımsız ve tarafsız performans değerlendirme kavramı, denetlenebilir, hasta odaklı, hesap verebilirliği fazla olan, verimli, iyileştirilebilir, sürdürülebilir ve erişilebilir sistemler meydana getirebilmek açısından büyük bir önem taşımaktadır. Performans değerlendirme sürecinin ana aktörlerinden biri, belirli performans göstergelerinin tanımlanabileceği ve sağlık sisteminde gereken alanlarda düzenli olarak kullanılabilecek bir yapının sunulması olmakla beraber, bu sürece netlik kazandırmak amacıyla performansın bütünsel bir görünümünü oluşturabilmek için konuyu odak noktası alan araştırmacılar tarafından, sağlık sisteminin çeşitli yapıtaşlarında farklı araç ve yöntemler kullanılarak çalışmalara devam edilmektedir. Sağlık hizmetlerinde performans değerlendirme, her ne kadar zor ve karmaşık bir süreç olsa da sistemin iyileştirilebilmesi-geliştirilebilmesi adına kritik bir önem taşımaktadır. Bu derleme çalışmasında, sağlık alanında 2011-2021 yılları arasında gerçekleştirilen çeşitli performans değerlendirme çalışmalarında kullanılan araç ve yöntemler incelenerek özetlenmiş, çalışmalarda öne çıkan kavramlar sınıflandırılmıştır. Çalışmanın sonucunda literatürdeki eksikliklerin gözlemlenmesiyle, gerçekleştirilen çalışmanın, mevcut literatüre katkı sağlaması amaçlanmıştır.

Keywords

• Sağlık sistemleri
• performans
• performans değerlendirme
• DÖK
• VZA
• ÇKKV

References

1. [1] M. A. Karadayı, Y. G. Gökmen, L. G. Kasap, ve H. Tozan, "Sağlıkta güncel simülasyon yaklaşımları: Bir derleme çalışması," Int. J. Adv. Eng. Pure Sci., c. 31, s. 1, ss. 1–21, 2019.
2. [2] J. F. Levesque and K. Sutherland, "Combining patient, clinical and system perspectives in assessing performance in healthcare: an integrated measurement framework," BMC Health Serv. Res., vol. 20, no. 1, pp. 1–14, 2020.
3. [3] H. Zare, M. Tavana, A. Mardani, S. Masoudian, and M. Kamali Saraji, "A hybrid data envelopment analysis and game theory model for performance measurement in healthcare," Health Care Manag. Sci., vol. 22, no. 3, pp. 475–488, 2019.
4. [4] B. Ö. Yilmaz, "Multi-criteria decision making ( MCDM ) applications in military healthcare field," Journal of Health Systems and Policies, vol. 2, no. 2, pp. 149–181, 2020.
5. [5] A. Jannati, V. Sadeghi, A. Imani, and M. Saadati, "Effective coverage as a new approach to health system performance assessment: a scoping review," BMC Health Serv. Res., vol. 18, no. 1, pp. 1–13, 2018.
6. [6] M. Karaman, H. Kuşçu Karatepe, ve F. N. Kuşçu, "Sağlık işletmelerinde performans değerlendirme ve ölçme yöntemleri hakkında bir derleme çalışması," Yönetim Ekon. Edeb. İslami ve Polit. Bilim. Derg., c. 4, s. 1, ss. 153–171, 2019.
7. [7] P. Pietro Biancone, S. Secinaro, V. Brescia, and D. Iannaci, "Redefining the conceptual framework for quality of care," Qual. - Access to Success, vol. 21, no. 174, pp. 40–48, 2020.
8. [8] R.-E. Caunic, "Frameworks and measures for health systems," SEA - Practical Application of Science, vol. VII, no. 21, pp. 205–213, 2019.

9. [9] P. R. Boscolo, G. Callea, O. Ciani, and R. Tarricone, "Measuring value in health care: a comparative analysis of value-based frameworks," Clin. Ther., vol. 42, no. 1, pp. 34–43, 2020.
10. [10] R. W. Dubois and K. Westrich, "As value assessment frameworks evolve, are they finally ready for prime time? ," Value Heal., vol. 22, no. 9, pp. 977–980, 2019.
11. [11] E. Kumah et al., "Frameworks for health systems performance assessment: how comprehensive is Ghana’s holistic assessment tool? ," Glob. Heal. Res. Policy, vol. 5, no. 1, pp. 1-12, 2020.
12. [12] C. M. Sørup, P. Jacobsen, and J. L. Forberg, "Evaluation of emergency department performance - a systematic review on recommended performance and quality-in-care measures," Scand. J. Trauma. Resusc. Emerg. Med., vol. 21, no. 1, pp. 1-14, 2013.
13. [13] M. Ghazisaeidi, R. Safdari, M. Torabi, M. Mirzaee, J. Farzi, and A. Goodini, "Development of performance dashboards in healthcare sector: key practical issues," Acta Inform. Medica, vol. 23, no. 5, pp. 317–321, 2015.
14. [14] R. Bangalore Sathyananda, A. De Rijk, U. Manjunath, A. Krumeich, and C. P. Van Schayck, "Primary health centres’ performance assessment measures in developing countries: review of the empirical literature," BMC Health Serv. Res., vol. 18, no. 1, pp. 1–13, 2018.
15. [15] M. H. Rahman, T. J. Tumpa, S. M. Ali, and S. K. Paul, "A grey approach to predicting healthcare performance," Meas. J. Int. Meas. Confed., vol. 134, no. October, pp. 307–325, 2019.
16. [16] G. Noto, I. Corazza, K. Kļaviņa, J. Lepiksone, and S. Nuti, "Health system performanceassessment in small countries: the case study of Latvia," Int. J. Health Plann. Manage., vol. 34, no. 4, pp. 1408–1422, 2019.
17. [17] B. Zaadoud, Y. Chbab, and A. Chaouch, "Do performance measurement models have any impact on primary health care? a systematic review," Int. J. Heal. Gov., vol. 25, no. 4, pp. 319–334, 2020.
18. [18] L. Carnut and P. C. Narvai, "A meta-summarization of qualitative findings about health systems performance evaluation models: conceptual problems and comparability limitations," Inq. (United States), vol. 57, no. 1, pp. 1-19, 2020.
19. [19] E. Carini et al., "Assessing hospital performance indicators. what dimensions? evidence from an umbrella review," BMC Health Serv. Res., vol. 20, no. 1, pp. 1–13, 2020.
20. [20] R. D. Dana, A. R. Dikananda, D. Sudrajat, A. Wanto, and F. Fasya, "Measurement of health service performance through machine learning using clustering techniques," J. Phys. Conf. Ser., vol. 1360, no. 1, pp. 012017, 2019.
21. [21] C. K. M. Lee, C. T. Y. Ru, C. L. Yeung, K. L. Choy, and W. H. Ip, "Analyze the healthcare service requirement using fuzzy QFD," Comput. Ind., vol. 74, no. 1, pp. 1–15, 2015.
22. [22] T. Düzcü, O. Yıldırım, H. Zülfikar, E. Yılmaz Alarçin, B. Sezginer, and H. Tozan, "The perceived service quality on patient : evidence from university hospital," Journal of Health Systems and Policies, vol. 1, no. 3, pp. 90–100, 2019.
23. [23] F. P. Salvatore, S. Fanelli, C. C. Donelli, and M. Milone, "Value-based health-care principles in health-care organizations," Int. J. Organ. Anal., vol. ahead-of-print no. ahead-of-print, 2020.
24. [24] E. E. Austin, B. Blakely, C. Tufanaru, A. Selwood, J. Braithwaite, and R. Clay-Williams, "Strategies to measure and improve emergency department performance: a scoping review," Scand. J. Trauma. Resusc. Emerg. Med., vol. 28, no. 1, pp. 1–14, 2020.
25. [25] M. S. Gok and E. Altındağ, "Analysis of the cost and efficiency relationship: experience in the Turkish pay for performance system," Eur. J. Heal. Econ., vol. 16, no. 5, pp. 459–469, 2015.
26. [26] C. C. Chen and S. H. Cheng, "Does pay-for-performance benefit patients with multiple chronic conditions? evidence from a universal coverage health care system," Health Policy Plan., vol. 31, no. 1, pp. 83–90, 2016.
27. [27] D. A. Cross, P. Nong, C. Harris-Lemak, G. R. Cohen, A. Linden, and J. Adler-Milstein, "Practice strategies to improve primary care for chronic disease patients under a pay-for-value program," Healthcare, vol. 7, no. 1, pp. 30–37, 2019.
28. [28] B. Martin, J. Jones, M. Miller, and R. Johnson-Koenke, "Health care professionals’ perceptions of pay-for-performance in practice: a qualitative metasynthesis," Inq. (United States), vol. 57, 2020.
29. [29] A. Spano, A. Aroni, V. Tagliagambe, E. Mallus, and B. Bellò, "Performance and expenditure in Italian public healthcare organizations: does expenditure influence performance? ," Public Money Manag., vol. 0, no. 0, pp. 1–8, 2020.
30. [30] D. O. Meltzer and J. W. Chung, "The population value of quality indicator reporting: A framework for prioritizing health care performance measures," Health Aff., vol. 33, no. 1, pp. 132–139, 2014.
31. [31] K. Khan and S. Ramachandran, "Conceptual framework for performance assessment: competency, competence and performance in the context of assessments in healthcare - deciphering the terminology," Med. Teach., vol. 34, no. 11, pp. 920–928, 2012.
32. [32] B. Marchal, T. Hoerée, V. C. Da Silveira, S. Van Belle, N. S. Prashanth, and G. Kegels, "Building on the EGIPPS performance assessment: the multipolar framework as a heuristic to tackle the complexity of performance of public service oriented health care organisations," BMC Public Health, vol. 14, no. 1, pp. 1–11, 2014.
33. [33] C. Reeve, J. Humphreys, and J. Wakerman, "A comprehensive health service evaluation and monitoring framework," Eval. Program Plann., vol. 53, no. 1, pp. 91–98, 2015.
34. [34] M. L. Mendu et al., "Measuring quality in kidney care: an evaluation of existing quality metrics and approach to facilitating improvements in care delivery," J. Am. Soc. Nephrol., vol. 31, no. 3, pp. 602–614, 2020.
35. [35] H. C. Liu and K. Itoh, "Conceptual framework for holistic dialysis management based on key performance indicators," Ther. Apher. Dial., vol. 17, no. 5, pp. 532–550, 2013.
36. [36] A. Traberg, P. Jacobsen, and N. M. Duthiers, "Advancing the use of performance evaluation in health care," J. Heal. Organ. Manag., vol. 28, no. 3, pp. 422–436, 2014.
37. [37] X. Gu and K. Itoh, "Performance measures for a dialysis setting," J. Ren. Care, vol. 44, no. 1, pp. 52–59, 2018.
38. [38] N. Perić, M. M. Hofmarcher, and J. Simon, "Headline indicators for monitoring the performance of health systems: findings from the european health systems indicator (euHS_I) survey," Arch. Public Heal., vol. 76, no. 1, pp. 1–17, 2018.
39. [39] B. V. Seixas, F. Dionne, T. Conte, and C. Mitton, "Assessing value in health care: using an interpretive classification system to understand existing practices based on a systematic review," BMC Health Serv. Res., vol. 19, no. 1, pp. 1-11, 2019.
40. [40] S. Agarwal et al., "A conceptual framework for measuring community health workforce performance within primary health care systems," Hum. Resour. Health, vol. 17, no. 1, pp. 1–20, 2019.
41. [41] H. Kim and B. Jeon, "Developing a framework for performance assessment of the public long-term care system in Korea: methodological and policy lessons," Heal. Res. Policy Syst., vol. 18, no. 1, pp. 1–11, 2020. [1] M. A. Karadayı, Y. G. Gökmen, L. G. Kasap, ve H. Tozan, "Sağlıkta güncel simülasyon yaklaşımları: Bir derleme çalışması," Int. J. Adv. Eng. Pure Sci., c. 31, s. 1, ss. 1–21, 2019.
42. [2] J. F. Levesque and K. Sutherland, "Combining patient, clinical and system perspectives in assessing performance in healthcare: an integrated measurement framework," BMC Health Serv. Res., vol. 20, no. 1, pp. 1–14, 2020.
43. [3] H. Zare, M. Tavana, A. Mardani, S. Masoudian, and M. Kamali Saraji, "A hybrid data envelopment analysis and game theory model for performance measurement in healthcare," Health Care Manag. Sci., vol. 22, no. 3, pp. 475–488, 2019.
44. [4] B. Ö. Yilmaz, "Multi-criteria decision making ( MCDM ) applications in military healthcare field," Journal of Health Systems and Policies, vol. 2, no. 2, pp. 149–181, 2020.
45. [5] A. Jannati, V. Sadeghi, A. Imani, and M. Saadati, "Effective coverage as a new approach to health system performance assessment: a scoping review," BMC Health Serv. Res., vol. 18, no. 1, pp. 1–13, 2018.
46. [6] M. Karaman, H. Kuşçu Karatepe, ve F. N. Kuşçu, "Sağlık işletmelerinde performans değerlendirme ve ölçme yöntemleri hakkında bir derleme çalışması," Yönetim Ekon. Edeb. İslami ve Polit. Bilim. Derg., c. 4, s. 1, ss. 153–171, 2019.
47. [7] P. Pietro Biancone, S. Secinaro, V. Brescia, and D. Iannaci, "Redefining the conceptual framework for quality of care," Qual. - Access to Success, vol. 21, no. 174, pp. 40–48, 2020.
48. [8] R.-E. Caunic, "Frameworks and measures for health systems," SEA - Practical Application of Science, vol. VII, no. 21, pp. 205–213, 2019.
49. [9] P. R. Boscolo, G. Callea, O. Ciani, and R. Tarricone, "Measuring value in health care: a comparative analysis of value-based frameworks," Clin. Ther., vol. 42, no. 1, pp. 34–43, 2020.
50. [10] R. W. Dubois and K. Westrich, "As value assessment frameworks evolve, are they finally ready for prime time? ," Value Heal., vol. 22, no. 9, pp. 977–980, 2019.
51. [11] E. Kumah et al., "Frameworks for health systems performance assessment: how comprehensive is Ghana’s holistic assessment tool? ," Glob. Heal. Res. Policy, vol. 5, no. 1, pp. 1-12, 2020.
52. [12] C. M. Sørup, P. Jacobsen, and J. L. Forberg, "Evaluation of emergency department performance - a systematic review on recommended performance and quality-in-care measures," Scand. J. Trauma. Resusc. Emerg. Med., vol. 21, no. 1, pp. 1-14, 2013.
53. [13] M. Ghazisaeidi, R. Safdari, M. Torabi, M. Mirzaee, J. Farzi, and A. Goodini, "Development of performance dashboards in healthcare sector: key practical issues," Acta Inform. Medica, vol. 23, no. 5, pp. 317–321, 2015.
54. [14] R. Bangalore Sathyananda, A. De Rijk, U. Manjunath, A. Krumeich, and C. P. Van Schayck, "Primary health centres’ performance assessment measures in developing countries: review of the empirical literature," BMC Health Serv. Res., vol. 18, no. 1, pp. 1–13, 2018.
55. [15] M. H. Rahman, T. J. Tumpa, S. M. Ali, and S. K. Paul, "A grey approach to predicting healthcare performance," Meas. J. Int. Meas. Confed., vol. 134, no. October, pp. 307–325, 2019.
56. [16] G. Noto, I. Corazza, K. Kļaviņa, J. Lepiksone, and S. Nuti, "Health system performanceassessment in small countries: the case study of Latvia," Int. J. Health Plann. Manage., vol. 34, no. 4, pp. 1408–1422, 2019.
57. [17] B. Zaadoud, Y. Chbab, and A. Chaouch, "Do performance measurement models have any impact on primary health care? a systematic review," Int. J. Heal. Gov., vol. 25, no. 4, pp. 319–334, 2020.
58. [18] L. Carnut and P. C. Narvai, "A meta-summarization of qualitative findings about health systems performance evaluation models: conceptual problems and comparability limitations," Inq. (United States), vol. 57, no. 1, pp. 1-19, 2020.
59. [19] E. Carini et al., "Assessing hospital performance indicators. what dimensions? evidence from an umbrella review," BMC Health Serv. Res., vol. 20, no. 1, pp. 1–13, 2020.
60. [20] R. D. Dana, A. R. Dikananda, D. Sudrajat, A. Wanto, and F. Fasya, "Measurement of health service performance through machine learning using clustering techniques," J. Phys. Conf. Ser., vol. 1360, no. 1, pp. 012017, 2019.
61. [21] C. K. M. Lee, C. T. Y. Ru, C. L. Yeung, K. L. Choy, and W. H. Ip, "Analyze the healthcare service requirement using fuzzy QFD," Comput. Ind., vol. 74, no. 1, pp. 1–15, 2015.
62. [22] T. Düzcü, O. Yıldırım, H. Zülfikar, E. Yılmaz Alarçin, B. Sezginer, and H. Tozan, "The perceived service quality on patient : evidence from university hospital," Journal of Health Systems and Policies, vol. 1, no. 3, pp. 90–100, 2019.
63. [23] F. P. Salvatore, S. Fanelli, C. C. Donelli, and M. Milone, "Value-based health-care principles in health-care organizations," Int. J. Organ. Anal., vol. ahead-of-print no. ahead-of-print, 2020.
64. [24] E. E. Austin, B. Blakely, C. Tufanaru, A. Selwood, J. Braithwaite, and R. Clay-Williams, "Strategies to measure and improve emergency department performance: a scoping review," Scand. J. Trauma. Resusc. Emerg. Med., vol. 28, no. 1, pp. 1–14, 2020.
65. [25] M. S. Gok and E. Altındağ, "Analysis of the cost and efficiency relationship: experience in the Turkish pay for performance system," Eur. J. Heal. Econ., vol. 16, no. 5, pp. 459–469, 2015.
66. [26] C. C. Chen and S. H. Cheng, "Does pay-for-performance benefit patients with multiple chronic conditions? evidence from a universal coverage health care system," Health Policy Plan., vol. 31, no. 1, pp. 83–90, 2016.
67. [27] D. A. Cross, P. Nong, C. Harris-Lemak, G. R. Cohen, A. Linden, and J. Adler-Milstein, "Practice strategies to improve primary care for chronic disease patients under a pay-for-value program," Healthcare, vol. 7, no. 1, pp. 30–37, 2019.
68. [28] B. Martin, J. Jones, M. Miller, and R. Johnson-Koenke, "Health care professionals’ perceptions of pay-for-performance in practice: a qualitative metasynthesis," Inq. (United States), vol. 57, 2020.
69. [29] A. Spano, A. Aroni, V. Tagliagambe, E. Mallus, and B. Bellò, "Performance and expenditure in Italian public healthcare organizations: does expenditure influence performance? ," Public Money Manag., vol. 0, no. 0, pp. 1–8, 2020.
70. [30] D. O. Meltzer and J. W. Chung, "The population value of quality indicator reporting: A framework for prioritizing health care performance measures," Health Aff., vol. 33, no. 1, pp. 132–139, 2014.
71. [31] K. Khan and S. Ramachandran, "Conceptual framework for performance assessment: competency, competence and performance in the context of assessments in healthcare - deciphering the terminology," Med. Teach., vol. 34, no. 11, pp. 920–928, 2012.
72. [32] B. Marchal, T. Hoerée, V. C. Da Silveira, S. Van Belle, N. S. Prashanth, and G. Kegels, "Building on the EGIPPS performance assessment: the multipolar framework as a heuristic to tackle the complexity of performance of public service oriented health care organisations," BMC Public Health, vol. 14, no. 1, pp. 1–11, 2014.
73. [33] C. Reeve, J. Humphreys, and J. Wakerman, "A comprehensive health service evaluation and monitoring framework," Eval. Program Plann., vol. 53, no. 1, pp. 91–98, 2015.
74. [34] M. L. Mendu et al., "Measuring quality in kidney care: an evaluation of existing quality metrics and approach to facilitating improvements in care delivery," J. Am. Soc. Nephrol., vol. 31, no. 3, pp. 602–614, 2020.
75. [35] H. C. Liu and K. Itoh, "Conceptual framework for holistic dialysis management based on key performance indicators," Ther. Apher. Dial., vol. 17, no. 5, pp. 532–550, 2013.
76. [36] A. Traberg, P. Jacobsen, and N. M. Duthiers, "Advancing the use of performance evaluation in health care," J. Heal. Organ. Manag., vol. 28, no. 3, pp. 422–436, 2014.
77. [37] X. Gu and K. Itoh, "Performance measures for a dialysis setting," J. Ren. Care, vol. 44, no. 1, pp. 52–59, 2018.
78. [38] N. Perić, M. M. Hofmarcher, and J. Simon, "Headline indicators for monitoring the performance of health systems: findings from the european health systems indicator (euHS_I) survey," Arch. Public Heal., vol. 76, no. 1, pp. 1–17, 2018.
79. [39] B. V. Seixas, F. Dionne, T. Conte, and C. Mitton, "Assessing value in health care: using an interpretive classification system to understand existing practices based on a systematic review," BMC Health Serv. Res., vol. 19, no. 1, pp. 1-11, 2019.
80. [40] S. Agarwal et al., "A conceptual framework for measuring community health workforce performance within primary health care systems," Hum. Resour. Health, vol. 17, no. 1, pp. 1–20, 2019.
81. [41] H. Kim and B. Jeon, "Developing a framework for performance assessment of the public long-term care system in Korea: methodological and policy lessons," Heal. Res. Policy Syst., vol. 18, no. 1, pp. 1–11, 2020.
82. [42] M. S. Patel, M. M. Davis, and M. L. Lypson, "The value framework: training residents to provide value-based care for their patients," J. Gen. Intern. Med., vol. 27, no. 9, pp. 1210–1214, 2012.
83. [43] R. N. Kamal, S. E. Lindsay, and S. L. Eppler, "Patients should define value in health care: a conceptual framework," J. Hand Surg. Am., vol. 43, no. 11, pp. 1030–1034, 2018.
84. [44] C. Modica, "The value transformation framework: an approach to value-based care in federally qualified health centers," J. Healthc. Qual., vol. 42, no. 2, pp. 106–112, 2020.
85. [45] E. Teisberg, S. Wallace, and S. O’Hara, "Defining and implementing value-based health care: a strategic framework," Acad. Med., vol. 95, no. 5, pp. 682–685, 2020.
86. [46] R. F. Ijntema, D. J. Barten, H. B. Duits, B. V. Tjemkes, and C. Veenhof, "A health care value framework for physical therapy primary health care organizations," Qual. Manag. Health Care, vol. 30, no. 1, pp. 27–35, 2021.
87. [47] M. Yıldız, Ö. Kethüda, ve M. Kurutkan, "Dengeli ölçüm kartının hastane işletmelerinde uygulanması: örnek olay çalışması," Atatürk Üniversitesi İktisadi ve İdari Bilimler Dergisi, c. 27, s. 4, ss. 274201390–274201409, 2013.
88. [48] S. Karasavuran, "Stratejik değerlendirme ve kontrol: dengeli puan kartının sağlık hizmetlerinde kullanımı," Hacettepe Sağlık İdaresi Derg., c. 16, s. 2, ss. 69–89, 2013.
89. [49] İ. G. Saraç, "Performans ölçümünde dengeli ölçüm kartı sisteminin kullanımı ve bir sağlık kuruluşunda uygulanması," Yüksek Lisans tezi, Hastane ve Sağlık Kuruluşları Yönetim Programı, Dokuz Eylül Üniversitesi, İzmir, Türkiye, 2011.
90. [50] Z. Lin, Z. Yu, and L. Zhang, "Performance outcomes of balanced scorecard application in hospital administration in China," China Econ. Rev., vol. 30, no. 1, pp. 1–15, 2014.
91. [51] N. F. M. Shukri and A. Ramli, "Organizational structure and performances of responsible Malaysian healthcare providers: a balanced scorecard perspective," Procedia Econ. Financ., vol. 28, no. April, pp. 202–212, 2015.
92. [52] O. Korkmaz, "Kurumsal performans karnesi ölçümü: üçüncü basamak sağlık kuruluşunda bir uygulama," Anemon Muş Alparslan Üniversitesi Sos. Bilim. Derg., c. 6, s. 6, ss. 1051–1061, 2018.
93. [53] F. Behrouzi and A. Ma’aram, "Identification and ranking of specific balanced scorecard performance measures for hospitals: a case study of private hospitals in the Klang Valley area, Malaysia," Int. J. Health Plann. Manage., vol. 34, no. 4, pp. 1364–1376, 2019.
94. [54] I. Cattinelli et al., "Use of self-organizing maps for balanced scorecard analysis to monitor the performance of dialysis clinic chains," Health Care Manag. Sci., vol. 15, no. 1, pp. 79–90, 2012.
95. [55] R. Gauld et al., "Scorecards for health system performance assessment: the New Zealand example," Health Policy (New. York)., vol. 103, no. 2–3, pp. 200–208, 2011.
96. [56] S. Falle, R. Rauter, S. Engert, and R. J. Baumgartner, "Sustainability management with the sustainability balanced scorecard in SMEs: findings from an Austrian case study," Sustain., vol. 8, no. 6, pp. 1-16, 2016.
97. [57] Ö. Doğan ve A. Tanç, "Konaklama işletmelerinde veri zarflama analizi yöntemiyle faaliyet denetimi: Kapadokya örneği," İktisadi ve İdari Bilim. Derg., c. 22, s. 1, ss. 239- 259, 2008.
98. [58] Ö. Güdük ve E. Önder, "Türkiye’deki hastanelerde veri zarflama analizi ile yapılan performans çalışmaları," Sağlıkta Kalite ve Performans Derg., c. 17, s. 2, ss. 49–95, 2018.
99. [59] H. Bircan, "Veri zarflama anali̇zi̇ ile Si̇vas ili̇ merkez sağlık ocakları etki̇nli̇ği̇ni̇n ölçülmesi̇," C.Ü. İktisadi ve İdari Bilim. Derg., c. 12, s. 1, ss. 331–347, 2011.
100. [60] D. Borisov, C. Cicea, and C. Turlea, "DEA model for assessing efficiency in providing health Care," Manag. Res. Pract., vol. 4, no. 1, pp. 5–18, 2012.
101. [61] J. Du, J. Wang, Y. Chen, S. Y. Chou, and J. Zhu, "Incorporating health outcomes in Pennsylvania hospital efficiency: an additive super-efficiency DEA approach," Ann. Oper. Res., vol. 221, no. 1, pp. 161–172, 2014.
102. [62] A. Aristovnik, "Regional performance measurement of healthcare systems in the EU: a non-parametric approach," Lex Localis, vol. 13, no. 3, pp. 579–593, 2015.
103. [63] V. Yiğit ve H. Esen, "Pabon Lasso modeli ve veri zarflama analizi ile hastanelerde performans ölçümü," SDÜ Sağlık Bilim. Derg., c. 8, s. 2, ss. 1–1, 2017.
104. [64] E. Önder ve C. Boz, "Oecd ülkeleri̇ni̇n sağlık si̇stemi̇ performanslarının değerlendi̇ri̇lmesi̇," Sos. Guvence, c. 0, s. 11, ss. 24–61, 2017.
105. [65] E. E. Karsak and M. A. Karadayi, "Imprecise DEA framework for evaluating health-care performance of districts," Kybernetes, vol. 46, no. 4, pp. 706–727, 2017.
106. [66] M. Gençtürk ve O. Şenol, "Veri zarflama analiziyle kamu hastaneleri birliklerinde verimlilik analizi," Süleyman Demirel Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, c.4, s. 29, ss. 265–286, 2017.
107. [67] R. Stefko, B. Gavurova, and K. Kocisova, "Healthcare efficiency assessment using DEA analysis in the Slovak Republic," Health Econ. Rev., vol. 8, no. 1, pp. 1-12, 2018.
108. [68] S. Alfiero, V. Brescia, and F. Bert, "Intellectual capital-based performance improvement: a study in healthcare sector," BMC Health Serv. Res., vol. 21, no. 1, pp. 1–16, 2021.
109. [69] H. Tozan, "Fuzzy AHP based decision support system for technology selection in abrasive water jet cutting processes," Tehnicki vjesnik-Technical gazette, 2011.
110. [70] T. Temuçin, H. Tozan, Ö. Vayvay, M. Harničárová, and J. Valíček, "A fuzzy based decision model for nontraditional machining process selection," Int. J. Adv. Manuf. Technol., vol. 70, no. 9–12, pp. 2275–2282, 2014.
111. [71] N. Öztürk, H. Tozan, and Ö. Vayvay, "A new decision model approach for health technology assessment and a case study for dialysis alternatives in Turkey," Int. J. Environ. Res. Public Health, vol. 17, no. 10, 2020.
112. [72] S. Vahdat, L. Hamzehgardeshi, S. Hessam, and Z. Hamzehgardeshi, "Patient involvement in health care decision making: a review," Iran. Red Crescent Med. J., vol. 16, no. 1, pp. 1–7, 2014.
113. [73] M. Karatas, I. Karacan, and H. Tozan, "An integrated multi-criteria decision making methodology for health technology assessment," Eur. J. Ind. Eng., vol. 12, no. 4, pp. 504–534, 2018.
114. [74] L. Afkham, F. Abdi, and A. Rashidi, "Evaluation of service quality by using fuzzy MCDM: a case study in Iranian health-care centers," Manag. Sci. Lett., vol. 2, no. 1, pp. 291–300, 2012.
115. [75] M. A. Karadayi and E. E. Karsak, "Fuzzy MCDM approach for health-care performance assessment in Istanbul," WMSCI 2014- 18th World Multi-Conference Syst. Cybern. Informatics, Proc., vol. 2, no. 1, pp. 228–233, 2014.
116. [76] H. Akdag, T. Kalayci, S. Karagöz, H. Zülfikar, and D. Giz, "The evaluation of hospital service quality by fuzzy MCDM," Appl. Soft Comput. J., vol. 23, no. 1, pp. 239–248, 2014.
117. [77] S. Jiang, Y. Li, P. Jiang, and L. Gan, "Evaluating the quality performance of reconstructive community public health service based on weighted TOPSIS method," Adv. Intell. Syst. Comput., vol. 502, no.1, pp. 113–123, 2017.
118. [78] S. P. Türkoğlu, "Avrupa ülkelerinin sağlık göstergelerinin TOPSIS yöntemi ile değerlendirilmesi," AİBÜ Sosyal Bilimler Enstitüsü Dergisi, c. 18, s. 1, ss.65-78, 2018.
119. [79] A. Kar, Ö. Özer ve K. Avcı, "Acil servislerin TOPSIS ve gri ilişkisel analiz yöntemleriyle değerlendirilmesi," Akad. Araştırmalar ve Çalışmalar Derg., c. 10, s. 19, ss. 442–459, 2018.
120. [80] A. Yiğit, "Türkiye’de eğitim ve araştırma hastaneleri performansının TOPSIS yöntemi ile analizi," Ankara Sağlık Bilim. Derg., c. 5989, s. 2146, ss. 72–85, 2019.
121. [81] İ. Yüksel, "Kamu hastanelerinde hekimlere yönelik performansa dayalı ek ödeme model önerisi 1 proposal for performance based supplementary payment model for physicians in public hospitals 1," Hacettepe Sağlık İdaresi Dergisi, vol. 23, no. 3, pp. 387–410, 2020.
122. [82] F. Baş, "Sağlik bakanliğina bağli eği̇ti̇m ve araştirma hastaneleri̇ performanslarinin çok kri̇terli̇ karar verme yöntemleri̇ kullanilarak değerlendi̇ri̇lmesi̇," Doktora tezi, Sağlık Yönetimi Anabilim Dalı, Hacettepe Üniversitesi, Ankara, Türkiye, 2021.
123. [83] E. Grigoroudis, E. Orfanoudaki, and C. Zopounidis, "Strategic performance measurement in a healthcare organisation: a multiple criteria approach based on balanced scorecard," Omega, vol. 40, no. 1, pp. 104–119, 2012.
124. [84] J. Bisbe and J. Barrubés, "The balanced scorecard as a management tool for assessing and monitoring strategy implementation in health care organizations," Rev. Española Cardiol. (English Ed., vol. 65, no. 10, pp. 919–927, 2012.
125. [85] Y. Chen, R. Friedman, and Tony Simons, "Development of balanced scorecard for healthcare using interpretive structural modeling and analytic network process," Manag. Audit. J., vol. 28, no. 2, pp. 2–3, 2014.
126. [86] H. Rahimi, Z. Kavosi, P. Shojaei, and E. Kharazmi, "Key performance indicators in hospital based on balanced scorecard model," J. Heal. Manag. Informatics, vol. 4, no. 1, pp. 17–24, 2016.
127. [87] N. Ö. Doğan ve S. Gencan, "Bulanık analitik hiyerarşi prosesi temelli balanced scorecard yöntemiyle sağlık sektöründe stratejik önceliklerin belirlenmesi determining the strategic priorities in the healthcare sector using the fuzzy analytic hierarchy process based balanced scorecard methodology," Atatürk Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, c. 21, s. 2, ss. 695–714, 2017.