INVESTIGATION OF THE EFFECTS OF USING AUGMENTED REALITY APPS ON STUDENTS' LEARNING ACHIEVEMENT AND MOTIVATION IN ENGINEERING DRAWING COURSES

: In engineering education Virtual Reality (VR) and Augmented Reality (AR) technologies have been recently adopted in the teaching of technical subjects showing good potential for visual interaction with the objects. This study aims to investigate the effect of using a specifically developed mobile AR Engineering Drawing (ED) application (Apps) for improving the cognitive learning levels of students and learning achievement. In this study, experimental design with pre-test and post-test control groups and interview were used to measure of participants’ success. For this purpose, a total of 30 participants involved in the experimental group used AR application for training, whereas another 20 participants without AR training were kept as a control group. The increase in performance between pre-test and post-test was 39.7% for the experimental group and 17.2% for the control group. The results indicated a 19.6% higher learning performance in the experimental group through the use of the AR application for training. This result shows the very important effect the mobile AR engineering drawing application had in terms of acquiring new knowledge and skills on the participants. The study also shows that teaching Engineering Drawing subjects using AR Apps significantly increases students' conceptual and cognitive comprehension levels.


INTRODUCTION
Nowadays, the quick integration of vocational education graduates into industry is of great importance for companies in terms of economic gain and production performance.Therefore, students should acquire knowledge and skills related to their professional fields during their education.For this, especially in the teaching of technical subjects and complex concepts, new technologies need to be integrated into education.(Emreli et al. 2019) There are concerns from industry about the decline in standards of production due to the lack of understanding of technical drawings standards (Huerta et al. 2019).This deficiency is due to the fact that the training given in high schools, vocational schools, and engineering faculties is not well supported with practical applications that can be used along traditional teaching methods for large groups to give an opportunity to students to quickly update practices or standards (Arslan & Uzaslan, 2017).
In AR, the interaction of the virtual objects with the real-life is important for the three dimensional perception since the virtual objects are clothed over the real image obtained from the camera as a covering.Recently AR technologies and Apss are replacing expensive laboratory equipment and the interest of students also increases due to the interaction with these new n technologies (Nesterov et al. 2017).AR technology has been used for educational purposes in areas such as giving three-dimensionality to two-dimensional drawings, giving cognitive training, maintenance/repair tasks, and tools and materials in engineering education (Somyürek 2014, Huerta et al. 2019).In this way, with the help of AR two-dimensional objects can be given a third dimension to easily understand the ED with limited time and tools in a classroom environment, the details on parts can be understood in a shorter time minimizing the wrong or incomplete learning.In their study, Balak and Kısa (2016) investigated the use of a smartphone-based augmented reality technology for Engineering Drawing (ED) education and emphasized that the engineering drawing course has a very critical importance for engineering students.In their study, Li et al. (2017) stated that mislearning and misinterpretation can be reduced through accurate and effective visualization using an AR platform that includes analysis of engineering data types and simulations.Huerta et al. (2019) developed an application for vocational and technical education using augmented reality technology.In this application, the rules of reading ED and giving tolerance are explained effectively.Within the scope of the study, new content and materials were developed to provide ED education using VR and AR environments as an open-source digital platform.AR improves the relationship between the perception of the spatial capabilities of engineering students, where students felt comfortable using AR environments and perceive the learning content and technology as convenient for their purpose.3D virtual models help engineering students perform their visualization tasks and encourage the development of their spatial abilities (Martin et al. 2010(Martin et al. , 2015)).Chin et al. (2019) investigated the utility of an ARbased mobile learning system developed to teach a liberal arts course.The results of the study showed that the students who had the opportunity to learn with the developed AR-based system demonstrated higher learning motivation and had better learning performance and comprehension abilities than those who worked with traditional teaching approach.
Higher Education institutions urge students to use actual real-life concepts, experiences, technologies and tools.Working in groups help students identify and use their shared knowledge to solve tasks or problems.As in all other fields, efforts to support engineering and design education with VR and AR applications have been accelerated in recent years (Kaufmann & Dünser, 2007;Seth et al., 2011;Hu & Xiong, 2005).Technical drawing and drafting subjects are generally difficult for engineering students but essential for industrial applications.In addition to understanding and describing the graphical representation of engineering objects, students need to understand the relationship between 3D objects, assembly processes, manufacturing techniques and their technical drawing representations.However, traditionally teaching hours are limited to fully cover and explain the representation of 3D geometries, therefore augmented reality platforms can be embedded into the teaching practices to support the teaching of this subject (Chen et al., 2011;Unver, 2006).Balak and Miman (2020) analysed the use of a mobile AR apps in ED course to evaluate its usefulness for engineering students that failed the course.The students who took the course a second time used the AR apps.The results indicated that the AR apps used in the course is useful in teaching the subjects of the course.This study has also shown that AR contributes greatly to the easier comprehension of parts that are quite complex to understand.In another study (Balak et al, 2018) a scale for favoring the application of virtual reality in technical drawing courses and consists of 6 items was developed.For four weeks of technical drawing course, 28 students selected randomly were subject to education in two virtual classes of 3 views and cross sections.Students replied to the 6-item scale and the data necessary to validate the scale and conduct reliability analysis was collected.Item analysis implies that the participants of the study have positive attitudes towards the use of VR apps in a technical drawing course.In a study, (Wang, et al. 2020) to have a close contact with the models and the views, a convenient AR app is proposed in the study.With the help of the AR app, virtual models and real views are combined well.The models can be moved and rotated by touching the screen.Some videos of model assembly and disassembly can be obtained when scanning the quick response QR code.According to the results of study it is convenient for students to use mobile phone to study ED effectively.
Studies on education and learning performance of different teaching materials, generally use control group experimental methods with pre-test and post-test.For example, Akçayır et al. ( 2016) investigated the effects of AR apps on university students' laboratory skills and attitudes using the pre-testpost-test quasi-experimental method with control group design.The results showed that using AR technology significantly increased the laboratory skills on students.In addition, Ürey and Çepni (2014) evaluated the effects of AR applications developed within the scope of extracurricular activities on the attitudes of science-based and interdisciplinary students towards Science and Technology courses in terms of various variables.In this study, a simple experimental method with pre-test and post-test design was used.Rich points were used in the analysis of the data and the results of the evaluation were analysed with SPSS software.T-test, ANOVA, Kruskal-Wallis and Mann-Whitney U test were used during the analysis.
This study aims to investigate the effect of using a specifically developed mobile AR Engineering Drawing App for improving the cognitive learning levels of students and learning achievement.

Design and Development of AR System
In this study, the effects of a developed original AR application (Kofoğlu et al, 2019) were used to evaluate students' learning achievement and motivation in ED courses.The study was developed under the project "Virtual and Augmented Reality in Design for Manufacture" conducted within the EU Erasmus+ Vocational Education Strategic Partnerships and led by the University of Bursa Uludağ, University of Huddersfield (UK) and Technical University of Sofia (Bulgaria).This project enabled a multidisciplinary team to use some of the newly emerging digital technologies (V/AR) in vocational and mechanical engineering education.Further project details can be obtained from www.vrindesign.org.The Research aimed to answer the following research questions by a comparison study done between two groups of students; one supported via AR application and the second supported only by conventional teaching methods.
 Differences in the learning performance,  Learning motivations,  Effect of AR in learning performance.
Initially, a need analysis with 320 people from different industrial and educational backgrounds was carried out to identify the areas of Engineering Drawings that may require support with the AR technologies.The survey included questions such as ED perception, knowledge and skill levels and expectation of ED education (Figure 1).Using the data from surveys AR and VR apps were developed.Users' perception was evaluated on the six subject areas as shown in Figure 2. In this research, only the effect of the mobile-based AR tool is evaluated (Figure 3).

Research Model
The framework of this study is to test the developed AR app in the form of pilot trainings and to evaluate the success of the trainings and the contribution to the competencies of the students by the peer assessment method.Experimental Pre-test-post-test control group designs are well suited to investigating effects of educational innovations and are common in educational research (Dugard & Todman, 1995).True Experimental design with pre-test and post-test control groups was used to establish the cause-effect relationship of the aforementioned AR app on students' perception and learning.True experimental designs are preferred for the consistency of analysis made with measurement and evaluation.For this purpose, theoretical exam and a questionnaire were done before and after training.Again, at the end of the study, interviews were held with the experimental group students to learn their views on AR apps.The evaluation involved an experimental group (supported by AR application and animations) and a control group (not supported by AR).
Pilot training was delivered during a month and with a total duration of 12 hours.Within the scope of the pilot training, a total of 12 hours of target oriented ED training was given to the experimental and control groups using the same place and accompanied by the same lecturer.It is a method that aims to teach the participants the educational attainments completely, by focusing directly on the desired subject headings, instead of the detailed content given on a semester-based basis, of the education content that is meant by target-oriented education.This method is especially preferred in internal trainings in companies and certification programs given within the framework of lifelong learning (Arslan& Uzaslan 2017).

Study Group
The study was realized on students studying at the department of Automotive Technology of the Vocational School of Technical Sciences.Initially, a total of 50 students were randomly selected for sampling, 30 of whom were trained using the AR app as support material (experimental group), and the remaining 20 (control group) were trained using only traditional methods.The experimental group consists of twenty-nine male and one female students.The control group consists of twenty male students.
The 50 student group was initially trained on the six identified topics using two-dimensional presentations conducted as teacher-centred activities.The same content was given by the same instructor to the 50 students, who were studying in the same program.Only the randomly selected group of 30 students received additional support from the AR application.Students within this selected group were enabled to download the application to their mobile phones and use it them individually.During the lesson, the related application images were projected onto the screen from a mobile phone, in case there was no possibility to download the application or there would be any system problems with their phones.After training, students were assessed and the effect of the AR training app evaluated.

Data Collection and Analyses
As mentioned before, to evaluate the effect of the AR application on the educational success and the contribution to the students' skills, a comparative competence analysis was conducted on both groups of students using an experimental measurement method.The performance measurement was done through a pre-test evaluation at the beginning of the course.The process was repeated as a post-test at the end of the course (Emreli, 2019).Finally, the results were analysed and evaluated using statistical software.The SPSS23 independent samples t-test (95% confidence interval) was used to compare the experimental and control group performance.
Likert scale questionnaires ranging from 1 (negative) to 5 (positive) consisted of 20 questions that were applied to the experimental and control groups via Google survey.Eight questions assessed the perception of technical concepts, seven were skill-based and five were behaviourmotivation related questions.The scale was designed in collaboration with academics with pedagogy and engineering background to evaluate the performance on education.Again, an interview consisting of 10 questions was conducted with 6 students from different levels according to the pre-test classical exam evaluations with the experimental group in the 3rd week.The obtained data were evaluated by subjecting them to numerical success, question-based questionnaire evaluation and content analysis.

RESULTS AND DISCUSSION
Comparisons of the pre-and post-test for the theoretical exam applied to both groups tested are given in Figure 4.It was observed that both groups were at the same level in the preliminary tests which included questions about the basic technical information for a drawing, the shapes and tolerances given on an Engineering Drawings.

Figure 4: Examination Pre-test Post-test comparisons
The analysis of the data collected showed an increased success rate of 481% in the experimental group and 362% for the control group.These results represent a theoretical learning performance 32% higher for the experimental group.The performance assessment results using 5-Likert scale of the study are given in Figure 5.The pre-test results were 2.92 for the experimental group and 2.91 for the control group showing similar results of pre-tests for theoretical test.In the post-tests, the experimental group results were found to be 4.08 and the control group as 3.41.The increase between pre-test and post-test was 39.7% in the experimental group and 17.2% in the control group.These results indicate 19.6% higher learning performance in the experimental group compared to the control group.Apart from the overall performance evaluation, the study included knowledge, skill and motivation sub-categories and question-based evaluations.In these assessments, it was observed that AR use increased 21.3% in technical knowledge, 20.58% in skill level, and 16.7% in behaviour-motivation (Emreli, 2019).This demonstrates a significant improvement in conceptual and cognitive comprehension levels on ED course participants through the improvement of knowledge and behaviours.
The SPSS23 independent samples t-test was used to compare the experiment and control group participant performance.Table 1 presents the independent samples t-test for pre-tests.The p value (p = .949>.05)obtained according to the data in Table 1 has shown that there is no significant difference in the levels of conceptual and cognitive understanding between the experimental and control groups before the experiment.On the other hand, the knowledge of the ED course after the experiment differed significantly (p= .000<.05) on the experimental and control groups as seen in Table 2. Consequently, it is possible to observe positive significant differences on the experimental group after ED training was received using the AR application.Therefore, the teaching and learning methods using AR have an important role in the learning performance of ED courses.
Overall, participants in the AR group conditions showed better learning than participants in the control group.The content analysis of interview showed that several factors affect the use of AR, and its impact on cognitive processes on participants and ED courses, including:  Motivation: The students who participated in the interview had not used VR&AR applications before.3D application makes it easier to focus, understand and increase students' learning motivation. Achievement and Ability: The application increased the students' ED success due to their ability to move from 2-D drawing to 3-D environment.Visual animation ability was also improved.The application was found to be successful by students  Student expectation: ED curriculum has recently seen major updates as a result of the integration of 3D tools (Unver, 2006) and widespread use of CAD-CAM software.Students now expect more integrated solutions that make teaching and learning easier and effective. Preferences: Students tend to prefer mobile devices due to their more accessible and practical applications.However, some students stated that VR glasses could also be used where more interactive contents can be generated.Students also stated that they preferred the use of this type of application during the lessons rather than for home use.

CONCLUSION
In this study, a previously developed AR application is evaluated for ED training for the individual learning needs of different levels, from technical high school to engineering faculty, and in particular in the machine manufacturing sectors.This research shows that the use of AR applications for teaching and learning ED has an overall positive impact that improves students' motivation and understanding.There are many aspects of AR technology that still need to be researched and developed, such as:  Technology, tools, update,  Content development, pedagogy,  Design and user experience,  Cost of development, accessibility, life of product, Many studies going on today is about the development of AR apps, but more research is still needed on pedagogy and user experience.Because in the near future, as technology continues to mature, less information will be available about the user experience and its pedagogical effects on students.The cost of developing these applications and maintaining support should also be considered.The findings on the performance and the contribution to individual's conceptual and cognitive learning levels indicated that the added value of the developed and evaluated AR application to learning performance is significant.
An overall positive feedback was received from the study where users used the developed AR content.The need and expectation for such applications are clearly seen in the content analysis outputs of the study.The expectation for the development of an interactive Engineering Drawing book that is integrated with mobile devices, that applications developed especially for mobile devices will be more useful, can be said as another important output.

Figure 1 :
Figure 1: Distribution of Groups Interviewed

Figure 2 :
Figure 2: Home screen of the mobile AR App

Figure 3 :
Figure 3: Sectioning screenshot of the mobile AR App

Figure 5 :
Figure 5:Performance measurement results(Unver et al, 2019) -test and post-test for theoretical exam

Table 1 . Independent samples t-test for pre-test results
Table 2 also presents independent samples t-test for post-tests.