A Qualitative Evaluation of a Non-Thesis Graduate Program of Human Resources Development in Education

Calismanin amaci: Bu calisma, Ankara’da bir devlet universitesinde Egitim Fakultesi bunyesinde faaliyet gosteren Egitimde Insan Kaynaklarini Gelistirme Tezsiz Yuksek Lisans Programini paydaslarin gorusleri dogrultusunda degerlendirmeyi amaclamaktadir. Materyal ve Yontem: Bu nitel degerlendirme calismasinda, arastirmacilar tarafindan gelistirilen bir anket ve yari yapilandirilmis bireysel gorusmeler yoluyla veri toplanmistir. Programa kayitli olan ogrencilere (n = 15) ve programdan mezun ogrencilerden erisilebilir olanlara (n = 60) anket uygulanmistir. Daha sonra amacli orneklem yontemi kullanilarak gorusme yapilacak katilimcilar belirlenmistir. Bu kapsamda, ogrenci (n = 5) ve mezunlardan (n = 20) gonullu olanlarla yari yapilandirilmis bireysel gorusmeler gerceklestirilmistir. Anket sonuclari, betimsel analizin ardindan frekans ve yuzdeler yoluyla paylasilmistir. Nitel verilerin analizinde ise icerik analizi yontemi kullanilmistir. Icerik analizi sonucu nitel verilerden anlamli parcalara kodlar atanmis ve birbiriyle iliskili olan kodlar bir araya getirilerek kategoriler olusturulmustur. Bunlar: 1) katilimcilari programi secmesinde etkili olan faktorler, 2) katilimcilarin programdan beklentilerinin karsilanma derecesi, 3) programin guclu yanlari, 4) program suresince karsilasilan zorluklar, 5) programin gelistirilmesi gereken yonleri seklinde listelenmistir. Bulgular: Bulgular, katilimcilarin programa kayit olmasindaki en onemli faktorlerin mesleki gelisim (n = 56) ve universitenin alandaki itibari (n = 56) seklinde siraladigini ortaya koymaktadir. Ayrica, katilimcilar, programin kisisel ilgilerini karsilamada (n = 54) ve mesleki gelisime katki anlaminda (n = 49) basarili oldugunu vurgulamistir. Bunlarin disinda, stajyerlik ve aksam derslerine yonelik belirtilen aksakliklara karsin, programin ogrencilere insan kaynaklari alaniyla ilgili gerekli bilgi ve beceri kazandirma konusunda basarili bulundugu sonucuna ulasilmistir. Onemli Vurgular: Bulgularin programin yeniden duzenlenmesinde katki saglamasi beklenmektedir. Bunun disinda, bu arastirma sonuclarinin, yuksekogretim kurumlarinda yer alan benzer programlarin gelistirilmesinde ve mevcut programlarin degerlendirilmesinde rehberlik etmesi hedeflenmistir. Son olarak, gelecekte, isverenlerin gorusmelere dâhil edilmesi ve programa kayitli ogrenci ve mezunlarin is yerlerinde gozlemler yapilmasi ile daha zengin sonuclara ulasilmasina olanak saglayacak calismalar yapilmasi ongorulmektedir.


INTRODUCTION
In the contemporary business world, the competitive global market has led to changes in business practices. Rapid changes in economy and technology and demands for more qualified employees have urged the foundation of Human Resources Development (HRD) departments which are now an essential component of empowering people in their own area of operation (Holton, & Trott, 1996). Nevertheless, what HRD is or what it ought to be is still a controversial issue. Built on a behavioristic approach, Nadler (1989) defines HRD as "series of organized activities conducted within a specific time and designed to produce behavioral change" (p. 3). In their prolific contribution, Swanson, Holton, and Holton (2001) adduce that HRD does not only deal with organizational development but also focuses on employee training and career development. On the other hand, premised on a more holistic view, Walton (1999) underlines the reciprocal relationships among global, national, organizational, and individual needs (p. 54). Despite some of the commonplaces in those definitions, there is not any consensus on the contested meanings and embedded practices of HRD that warrants empirical research. In other words, in the scholarly literature, there are some deficiencies in addressing the real business issues, and meeting the needs of both researchers and practitioners (Kuchinke, 2003).
Having technical skills to instruct others and knowledge on curriculum, assessment, facilitating learning, etc., these specialists play a critical role in HRD (Walter, 2006). Following the increasing demand for HRD specialists, the number of higher education institutions that implement such academic programs has increased (Grubb, Hemby, & Conerly-Stewart, 1998;Kuchinke, 2002;Kuchinke, Fu, & Oh, 2006). Therefore, students with different academic backgrounds have enrolled in HRD programs to become a member of this bread and butter activity (Gray, 1997, p. 80). Nevertheless, the programs are not firmly structured with regards to curriculum and institutional affiliation (Kuchinke, 2000) ruining the disciplinary base of the field (McLean, 1998). As being an emerging, as well as contextual and complex field in nature, systematic study of HRD academic programs is demanding (Kuchinke, 2003). Nonetheless, there is a dearth of literature about HRD programs, their institutional arrangements, and curriculum (Kuchinke, 2001).
Bearing on the demand for program evaluation studies in the field, it can be concluded that program evaluation is a quintessential aspect of addressing problems with regards to student learning, deciding the effectiveness of a teaching method and continuum or termination of a program (Doğan, 1997;Marsh & Willis, 2003). In other words, program evaluation enables researchers to collect evidence on strengths and weaknesses of the program, feasibility of the program, and success in the workforce (Oliva, 2009); for this reason, program evaluation is a critical component of improving or maintaining the quality of any education program (Tyler, 1949). In this respect, when the literature is reviewed, it is seen that, albeit limited, some evaluation studies focus specifically on the HRD academic programs. To illustrate, in his theoretical study,  proposed a new evaluation model as opposed to Kirkpatrick's (1995) four-level evaluation model. Moreover, in his comparative research, Kuchinke (2003) examined the institutional and curricular characteristics of HRD master's programs in the UK and the US. The findings of the study demonstrated that there is a lack of standardization among HRD programs in the US in terms of coordination, accreditation, or certification. In addition, it was claimed that the training offered in those programs is vocational-based, as is the case for the UK. Contrary to the aforementioned studies, the literature in Turkey still needs to be elucidated. The studies in the national literature focus on human resources management and employees' motivation, performance, or institutional loyalty (Bingöl, 2010;Çalışkan, 2010;Çöl, 2004;Doğan, & Demiral, 2008;Öğüt, Akgemci, & Demirsel, 2004). In addition, most of the theses were published within the Department of Business, Labor Economics and Industrial Relations, and others (e.g. Political Science and Public Administration, Psychology, Tourism Management, Public health, etc.). In contrast, the percentage of those theses published in the field of Educational Sciences is only 1% (Benligiray, 2009).
Succinctly, some aspects of HRD education still need scholarly scrutiny. In this respect, Kuchinke's (2001) salient study which resulted in a research agenda for the field has opened rooms to be investigated for the advancement of the field. The author elucidated three main research topics to be studied namely, the subject matter of HRD, institutional characteristics, arrangements, and linkages, and the effectiveness of HRD education. Grounded on these findings, this paper aims to examine the effectiveness of a non-thesis master's program -Human Resources Development in Education (HRDE)-offered in one of the leading Englishmedium universities of Turkey. The systematic study of this program is critical since unlike other graduate programs located in Schools of Business and Management; this program is offered in the School of Education. Besides, the program has not been evaluated since it was established in 2001; therefore, in order to provide insights for the continuum of the program, it is important to evaluate the effectiveness of the program in various aspects. The purpose of this study, then, is to provide information about students' expectations, the problems encountered in the program, and the strengths as well as the needed improvements of the program which will, in turn, increase the quality of the program based on perceptions of the stakeholders.

METHOD
This study is designed as a case study that aimed to evaluate the effectiveness of a non-thesis graduate program, Human Resources Development in Education (HRDE), at an English-medium public university in Ankara. The reason for selecting the case study as the research method is twofold. First, the program is unique in that contrary to the similar programs offered within business faculties, the program under investigation is housed at the education faculty. Second, the program has not been evaluated since the time it was founded in 2001.
In this study, the program implementation is described regarding the perceptions of students and faculty through process evaluation, and the information about program effectiveness is gathered through outcome evaluation with respect to the perceptions of graduates and students on what they learned through the program and whether they can implement the skills and knowledge they developed in their workplace.
The main research question of this study was "How do the participants (students, graduates, and instructors) perceive the effectiveness of the HRDE program?" and the following questions guided the present research.

The Context of the Study
The program under investigation is a non-thesis master's program housed at the Educational Sciences Department of an English-medium public university in Ankara. The program has been accepting students from various departments since the 2001-2002 academic year. Since then, almost 200 students have graduated from the program and most of them have started working as HR specialists at recognized companies. As graduation requirements, the students have to complete six must and four elective courses, as well as an internship in the last semester of the program. The courses include theoretical and practical knowledge from different disciplines such as Administration, Curriculum Development and Instruction, and Assessment and Counseling. The program mainly aims to equip its participants with the necessary knowledge and skills in the field of HRDE. People who are currently working in the field of HRD yet seek to empower their capabilities and have career plans in the field of HRDE can apply for the program.

Participants of the Study
The current students, program instructors, and graduate students of the program HRDE contributed to the evaluation of the program. Information on program effectiveness was gathered from the current and graduate students who gain benefits from the program and are primarily affected by outcomes of the program, as well as the faculty members. In this regard, the graduate students (n = 60) and the current students (n = 15) were administered a questionnaire developed by the researchers. For the qualitative part, five current students (nfemale = 3, and nmale = 2) volunteered to be interviewed, and 20 volunteer graduate students, who were purposely selected considering their graduation year, place of work, and position at work were interviewed (nfemale = 16 and nmale = 4).
The instructors (nfemale = 2, and nmale = 2) were offering must courses in the program. All of them worked at the Department of Educational Sciences: two from the Curriculum and Instruction program; one from the Guidance and Psychological Counseling program; one from the Educational Administration and Planning program.

Data Collection Tools and Procedures
In this study, the data were collected through a questionnaire developed by the researchers (see Figure 1 for the development of the questionnaire) and face-to-face semi-structured interviews.
For the development of the questionnaire, initially, an item pool was determined based on an extensive literature review in accordance with the purpose of the study. Following the preparation of the item pool for each part, two professors who are experts in test construction were consulted on their feedback on the format, content, appearance, and appropriateness of the items. Then, the necessary changes and revisions were made based on their suggestions. For instance, a dimension that intended to portray the strengths and weaknesses of the program was merged into the other dimensions, and a dimension that aimed to depict the overall perceptions of the participants about the effectiveness of the program was added in the questionnaire. The final version of the questionnaire mainly consisted of two parts. The first part included items on demographic information. In the second part, there were five dimensions about perceptions of the participants on several aspects of the program: (1) participants' preferences to the program (15 items); (2) participants' expectations from the program (14 items); (3) goals and outcomes of the program (13 items); (4) difficulties met during the enrollment in the program (11 items), and (5) overall opinion with regards to effectiveness of the program (25 items). For the first dimension, the participants were expected to respond to items about the factors that affect their preferences for the program on a four-point scale ranging from "very ineffective (1)" to "very effective (4)". For the other parts of the questionnaire, the participants were expected to rate each item on a four-point scale ranging from "completely disagree (1)" to "completely agree (4)". Next, before administering the questionnaire necessary permissions were taken. The questionnaire was administered to graduate students through an online program. Out of 197 graduate students, 98 graduate students were reached and informed about the study via e-mail. Eventually, 61.24 % of graduate students (n = 60) agreed to complete the questionnaire. Besides, the paper version of the questionnaire was administered to the students currently enrolled in the program (n = 15). Complementing the questionnaire, semi-structured interviews (see Figure 2 for details about the interview forms) were conducted vis-a-vis by the researchers about the effectiveness of the program. Interview protocols were developed by the researchers to obtain more detailed and comprehensive information about the program. Following, two professors who were experts in the field of qualitative research were asked for their feedback. Based on their feedback, some changes in the wording of the questions were made. For instance, the question "What are the drives behind your preference of this program?" was added instead of asking "Why did you enroll in this program?" Then, the final version of the protocols was given. The participants (faculty members, current students, and graduate students) were contacted by the researchers and invited to participate in the interviews.

Figure 2. Interview schedules
Semi-structured interviews were conducted in places where the participants preferred to be interviewed. Interviews lasted around 30 to 50 minutes and all of them were audiotaped. Then, all the oral data were transcribed verbatim in the original interview language that was Turkish.

Data Analysis Procedures
The quantitative data which were obtained through the questionnaire were analyzed descriptively using SPSS IBM 24. The descriptive analysis included frequencies and percentages. Besides, the qualitative data which was gathered through semistructured interviews were subjected to content analysis. In this context, initially, transcription of all of the interviews was done. Then, two of the researchers read all the transcribed data. Initial codes were assigned to meaningful segments, and those codes were brought together to generate themes considering the research questions and the related literature. Next, two professors who were experts in the qualitative research were consulted for their feedback on the generated themes. After negotiations, the final version of the codes and themes were determined as (1) the factors that affect participants' preferences to the program; (2) the degree to which participants' expectations from the program were met; (3) strengths of the program; (4) the difficulties encountered during the program; and (5) the aspects of the program that need improvement.

Trustworthiness and Limitations of the Study
According to Yin (2003), case studies have limitations regarding potential biases for several reasons such as lack of control for outside variables, lack of randomization, lack of generalizability, and the problems derived from establishing reliability. In order to control these limitations and ensure the trustworthiness of the study, the researchers employed the following strategies.
First, a thorough picture of the HRDE program was portrayed to increase the likelihood of the transferability of the findings of this study into similar cases. Also, triangulation of data sources and triangulation of data collection methods were employed to increase the credibility of the study (Miles & Huberman, 1994). Besides, the inter-coder agreement was calculated to provide a thorough depiction of the emerged data. In this regard, two of the researchers independently created codes and themes from the transcriptions of randomly selected two interviews and the codes and themes they generated were compared to check the consistency. As a result, it was revealed that two independent coders agreed on the coding of the given content with the level of agreement of 90%, as calculated with the Miles and Huberman's agreement formula, which is accepted as highly reliable (Miles, & Hu-berman, 1994). For member checking, on the other hand, the researchers asked the participants to check the transcripts of their interviews to enhance the accuracy of the transcription of the interviews after transcribing the audio recordings.

FINDINGS
Findings of this study are reported under five major subtitles: (1) the factors that affect participants' preferences to the program; (2) the degree to which participants' expectations from the program were met; (3) strengths of the program; (4) the difficulties encountered during the program; and (5) the aspects of the program that need improvement.

The Factors that Affect Participants' Preferences to the Program
The current and graduate students were asked to identify the factors that influence their preferences for the program. Descriptive statistics revealed that the factors which affected the preferences of the participants for the program were mainly professional development (n = 56), the reputation of the university (n = 56), personal development and interest (n = 53), getting a master's degree diploma (n = 50), the reputation of the department (n = 47), and following the scientific developments and trends in the field (n = 46). On the other hand, the demands of the employers (n = 9), an opportunity for Ph.D. education (n = 13), a rise in salary (n = 20), and being related to the field of the undergraduate program (n = 22) were the factors that participants considered as less effective on their preferences for the program.
The interviews revealed similar findings. One of the graduate students, who works at the HR department of a private company, stressed the reputation of the university as a major factor affecting his preferences for the program by stating that: …I wanted a career in the field of HR. I searched for master's programs in this field and decided to study at this university which is well-known for its inspiring culture and qualified instructors. In addition, due to the medium of instruction, I can easily follow the international literature.
Also, a student who worked as an instructor reported why she chose this program as: I will have a master-degree when I graduate from the program and will be able to certificate my knowledge and skills I gained through this program. And, being a graduate of X University will help me go one step further compared to my colleges.

The Degree to which Participants' Expectations From the Program Are Met
The students and graduates of the program stakeholders reflected their perceptions on to what degree the program meets their expectations. In most of the cases, the program met their expectations regarding personal interest (n = 54), broadened perspective on issues of the workplace (n = 51), professional development (n = 49), scientific developments and trends in the field (n = 48), creating different solutions for the work problems (n = 46), and developing social and business network (n = 43). On the other hand, less dominant interest specified by the participants was related to meeting employee's expectations (n = 30), getting a promotion (n = 24), and a rise in salary (n = 12).
As complementary, one of the graduates of the program who had been working as an HR expert for 10 years reported the benefits of the program in his professional development by stating that:

First of all, one of the most important dimensions of HR is education, which is already a strong aspect of this program…I have the theoretical information about the performance management system, thanks to the program. Moreover, I learned useful information about job interviews. Contrary to my colleagues who ask ordinary questions in job interviews, I can reach different perspectives. Therefore, I can say that the opportunities the program provided us were beyond my expectations.
Another graduate student who worked as an HR expert mentioned the advantages of the program concerning offering new job opportunities by saying that: …Throughout the program, we learned what / how we should make research in the field of HR. I completed my internship at a recognized company and they were very satisfied with me. Then, they offered me a position and I started to work as a human resource expert. That's the major contribution of the program.
The instructors of the program, on the other hand, emphasized the benefits of the program to the students and focused on several benefits of the program in accordance with the professional development of the students: (a) integrating theory and practice, (b) applying theoretical knowledge in the work-life, (c) gaining different points of view regarding planning and evaluating training, (d) offering job opportunities and (e) following up with students' professional development in their work life.

Strengths of the Program
In the questionnaire, the graduates and current students were asked about their perceptions about the goals and outcomes of the program. The results showed that with regards to strengths of the program, more than half of the participants (n = 53) found the program relatively successful in relation to the attainment of the two major program goals: (a) The program provides career opportunities in the field of HRD in the prestigious institutions and organizations (n = 48) and (b) The program prepares practitioners who can be a leader in developing various educational activities about human resource development such as curriculum development, instruction, and assessment (n = 48).
Accordingly, a graduate student who works as a training and development specialist at a human resource department of a private organization reported the most powerful feature of the program as providing job opportunities by stating that: …After graduating from the program, I felt more secured and confident in my professional life. Let me put it this way: I can easily find jobs. When I do not want to work at that company I quit the job then I can easily find another job. I am the one who set the working conditions when I go for a job interview.
Another graduate student made a strong emphasis on the impact of the program on his performance in various areas of education such as curriculum development and instruction and assessment by asserting that: Thanks to the knowledge and skills which I gained through the program, when we need to make changes in the institution, I can better plan and evaluate the change process, now. Also, report! I mean, through providing a report, we inform others about the findings and drives behind making such changes which helped us to create a peaceful working environment here.
Other strengths of the program were displayed in Table 1. In this regard, a graduate student working as an HR expert revealed that the program integrated theory and experiential training for its students, which makes their work easier: We mainly covered the topics through integrating theory and practice. For example, we developed a survey and administered it. Also, we engaged in an evaluation process and presented our findings. I felt like we were running a project at a company.
Furthermore, a graduate student who works as a specialist in the field of HR stressed the major difference of the program from similar programs as the program is helpful in terms of developing the key competencies essential for current and future practitioners in the training and development field by saying: Companies do not hire a curriculum expert, a measurement evaluation expert, an e-learning designer because they expect that an HR specialist can do all. This is what HRDE provides us.
Next, the stakeholder's overall perceptions of the program were displayed in Table 2. The instructors of the program also reported similar findings. They specified three types of skills that graduate students gained during the program: (1) educational skills, (2) research skills, and (3) personal skills. Educational skills include i) planning educational experiences, ii) developing learning activities and materials, iii) implementing learning activities, and iv) evaluating educational experiences. Research skills, on the other hand, refer to i) developing data collection tools, ii) conducting needs assessment, iii) collecting data, iv) analyzing data, and v) reporting. Lastly, personal skills are listed as i) self-awareness, ii) communication, iii) cooperation, iv) network-ing, v) self-discipline, vi) presentation skills, and lastly, vii) computer skills.
As another strength of the program, the faculty underlined the role of the program in increasing the reputation of the program, side income for the faculty, and professional development of the faculty, particularly in meeting the needs of diverse students.
To sum up, while the program appeared to accomplish its goals and outcomes, it also provided students with the opportunity for professional and personal development. Put differently, the participants reported that the program was successful to meet the needs and expectations of the diverse students. Moreover, qualified faculty members and their positive approach to students were found as other strengths of the program. Also, the updated content of the courses, English-medium instruction, and the opportunities for both individual and group works were revealed as strengths of the program.

Difficulties Encountered During the Program
When the challenges which current and graduate students experienced during the program were sought, the most difficult aspects were listed as high program fee (n = 28), overloaded courses (n = 18), the limited number of elective courses (n = 16), late course hours (n = 14), and difficulties in access to the course material (n = 14).
The interviews yielded identical results. The participants uttered that they had difficulties in i) transportation to the faculty; ii) adapting to long course hours, iii) selecting courses, iv) working on scenarios rather than real cases for the term projects, and v) communicating with their peers and the instructors. In this regard, one of the early graduates of the program who was working in the HR department of a company mentioned lack of real cases and problems for the term projects as a major weakness of the program by stating that: …the focus of the program was mostly related to education as expected because it is the program in the Faculty of Education. But, as an HRD program, it lacked the practical information related to the field other than the educational aspect. I graduated from the Business Department and when I started the program, I was confused and did not know what to do at first. I expected more information and courses on work-life and real cases in the field of HR.

The Aspects of the Program that Need Improvement
When the aspects of the program that need improvement were investigated, the participants mostly emphasized i) lack of emphasis on the management dimension of the field of HRD; ii) lack of elective courses on personnel selection and the organizational dimension of the field; iii) lack of instructors, experts in the related fields of business and management. In addition, they uttered problems with regards to the advertisement of the program, use of technology in classes, visibility of the program in career fairs, assistance to participate in academic meetings, and flow of the classes. One of the recent graduates of the program claimed that: …For instance, most of the companies develop projects in the field of human resources. I am also working on such a project. I wish that students from this department who wants to be a human re-source expert can work with me in this project. This enables students to have work experiences and I can work with someone interested in the field, as well. The program lacks this kind of network, I think. The program needs to provide communication among graduate and current students.
Another graduate, who works as an HR manager at a renowned company, remarked that the visibility of the program should be increased to reach more people who seek to advance their career opportunities by attending such programs. She continued as: I learned the existence of such a program from a friend of mine. He encouraged me to attend this program.
When I started working, I heard many people saying to me that they did not know that there was a graduate program offered at the Faculty of Education at X (the name of the university). The department should do something to increase the visibility of the program. They can hire advertisement boards, or create social media accounts to share announcements about the program. I think, more and more people will apply for the program, then.
As complementary, according to the faculty members, the program needs to be improved in terms of the content of the courses; integrating theory and practice; motivation of the faculty to teach; revising the program on a systematic basis; communication among faculty members; the internship; orientation program; application and admission criteria. An excerpt from the interview with a professor at the Department of Curriculum and Instruction summarizes the faculty perceptions as: We were very excited when we opened the program. However, we lost our motivation over time. Evening classes made our faculty unwilling to teach. Besides, the student profile changed. We had very talented and motivated students before, but now we have difficulties in motivating our students. Most of them are working; therefore, they are tired when they come to the class. They lose enthusiasm. We should do something innovative to re-energize the program.
Another faculty from the Educational Administration and Planning Department, on the other hand, depicted the problems regarding the internships as: We control how students complete the internship. They are expected to provide a report about the project they completed during the internship. Contrary to the majority of the students who complete all the work as required, some of them even do not go to the company they are supposed to complete the internship. They just provide us a copy of the signs of the company owner that certifies his/her internship. With those signs at hand, we cannot do anything.

DISCUSSION
This study intends to situate the effectiveness of a non-thesis graduate program, Human resources Development in Education, based on the perceptions of the stakeholders. Overall findings indicate that despite some of the aspects that need improvement, the program is effective in equipping the students with the necessary skills and knowledge the field requires.
The field of HRD has grown as the contemporary market changed. The companies have been investing in HR departments for organizational development and training of their personnel. Accordingly, higher education institutions have started offering academic programs to train individuals who seek a career opportunity in the field of HR (Kuchinke, 2001). Adding to the literature, our findings reveal that the main drive behind enrolling in the HRDE program was professional development. Besides, as also reported by Kuckinke (2004), the other mostly stat-ed factor was the reputation of the university. The students and the graduates noted that studying at a recognized university enabled them to find a job easily.
Next, the participants were asked about their perceptions about the effectiveness of the program. Although most of the participants alleged that the program is effective in equipping students with the necessary knowledge and skills, some students criticize too much emphasis on education rather than business and organization dimensions since the faculty members are specialized in the field of education. Indeed, this is a problem of all HRD programs housed in schools of education, despite the increased stability and sustainability of these programs (Kuchinke, 2004).
Furthermore, HRD education is provided by academic programs at higher education institutions and non-academic certification programs. This duality causes conflicts with regard to the content and structure of those programs. With regards to curricular content of these programs, it is concluded that a multi-disciplinary approach was utilized including courses HRD, Designing Training Programs, Group Dynamics, Organization Development, Consulting Skills, Personnel Methods, Organizational Behavior, Organizational Communication, Labor and Industrial Relations, Sociology of Work, and Career Development (Watkins, & Marsick, 2016, p. 471). Kuchinke (2001) examined 55 HRD graduate programs and also collected information on program administrates. He reported that the faculty members ranked instructional design and delivery, evaluation, adult learning, and needs analysis as the most important courses. The program under investigation for this study offers similar courses to its trainees. According to the participants, the content of the courses meets the program needs and is appropriate for the work-life. However, as some of the participants disclosed, albeit there are both theoretical and practical courses in the program, there is a need for elective courses on personnel selection and organizational dimension.
Moreover, graduates and students of HRDE claimed that they developed research, information management, communication, collaboration, and computer skills throughout the program. Besides, some of the participants stated that there are also side effects of enrolling in this program like developing better presentation skills, language skills, and critical thinking skills. Faculty members of the program, on the other hand, grouped the skills developed through the program as research skills (developing data collection tools, needs assessment, collecting data, analyzing data, reporting), personal skills (self-awareness, communication, and cooperation, networking, self-discipline, presentation skills, computer skills) and educational skills (planning educational experiences, developing learning activities and materials, implementing learning activities, evaluating educational experiences). In this regard, skills developed through the HRD programs have been a contested issue in the literature. In their follow-up study, Dare and Leach (1999) surveyed faculty members who teach at HRD departments. Skills such as writing, computer, informationsearch, visioning, data reduction, and research were found to yield increased importance compared to the previous study conducted by Leach (1993).
Given the knowledge and skills developed through the HRDE program, almost all of the participants mentioned the effectiveness of the program in regards to their personal and professional development, meeting personal needs and employers' expectations, being in line with the work-life, and developing a network. Despite the mentioned strengths of the program, the participants, however, also recognized several difficulties and improvements related to the program. Among them were (a) fewer opportunities for the internship; (b) relatively late class hours; (c) education-focused course contents, (d) lack of emphasis on the management dimension of the field of HRD, (e) lack of elective courses on personnel selection and the organizational dimension of the field, (f) lack of instructors, experts in the related fields of business and management, (g) gap between theory and practice and (h) lack of tracking alumni. Some of these deficiencies regarding course contents, business and management dimensions, theory and practice, and competencies are also mentioned in previous scholarly works in the field (e.g. Dare, & Leach, 1999;Kuchinke, 2002;Tranfield, & Denyer, 2004).

SUGGESTIONS
This program evaluation study has provided a body of useful data that has served to inform administrators and faculty members for its improvement to meet the ever-changing needs and expectations of the stakeholders. First, the strengths of the program and the parts which need improvements were classified by the stakeholders. These findings, therefore, might provide a basis for the improvement of the program. In addition, the findings revealed that the visibility of the program should be ensured by the advertisement of the program through social media and other online platforms and attending the academic conferences, as well as business expositions. Besides, cooperation with other departments such as Business and Administration to build a more interdisciplinary, thus holistic, understanding of Human Resources Development.
Second, the findings might be useful also for those who offer similar programs and seek to conduct a program evaluation study.
Third, faculty members who want to develop a similar program in their departments can benefit from the findings of the study.
Last, the findings of the study were mainly based on the perceptions of current students and graduates. Further research might be conducted to include perceptions of the employers about the performance of the students and graduates of the program. Also, observations might be conducted in the workplace of the program graduates and students. This method might enable researchers to acquire enriched information that cannot be easily accessed by a survey questionnaire. By combining interview and observation techniques, new insights and further understanding of specific information can be generated.

Ethics Committee Approval Information
All procedures performed in this study involving human participants were in accordance with the ethical standards of the institutional research committee at Middle East Technical University. The data were conducted before 2019; therefore, ethics committee approval was not required. The authors declare that they have no conflict of interest.