Development of a Computerized Adaptive Version of the Turkish Driving Licence Exam

Abstract

This study tested the applicability of the theoretical Examination for Candidates of Driving License (ECODL) in Turkey as a computerized adaptive test (CAT). Firstly, various simulation conditions were tested for the live CAT through an item response theory-based calibrated item bank. The application of the simulated CAT was based on data from e-exams administered by the Ministry of National Education (MoNE). Results of the first stage of the study were used to determine the rules for starting, continuing, and terminating the live CAT exam for ECODL. Secondly, the live CAT exam was applied according to the results of the simulation. Candidate drivers (n = 280) who had taken the ECODL as an e-test participated in the second stage. Thirdly, the opinions of the individuals who took the computer-based test towards the computer-based testing application were mapped. In the termination rule of the CAT-based ECODL, testing with a fixed number of questions yielded the smallest estimated measurement error. We also found that when ECODL was implemented as CAT, it could reliably differentiate among testers in terms of competence of theoretical knowledge of driving and provide basis for accurate decisions regarding their proficiency. According to the findings obtained on the candidates’ opinions on the computer-based testing application, it was seen that they considered computer-based application more practical an easier in terms of testing.

Keywords

• Computerized adaptive testing
• Driving licence exam
• Hybrid simulation

References

1. AngloInfo, The Global Expat Network Berlin. (2015). Taking a German driving test. Retrieved May 24, 2017, from http://berlin.angloinfo.com/information/transport/driving licences/the-driving-test
2. Balkız, C. (1999). The effects of traffic education on the prevention of traffic accidents. (Unpublished master thesis). Gazi University, Graduate School of Natural and Applied Sciences, Ankara.
3. Bozkurt, M (2003). Evaluation of the first aid knowledge of driving nominees at the begining and at the end of driving course in Ankara. (Unpublished master thesis). Gazi University, Graduate School of Natural and Applied Sciences, Ankara.
4. Bugbee, A. C. & Bernt, F. M. (1990). Testing by computer: Findings in six years of use 1982–1988. Journal of Research on Computing in Education, 23(1), 87 100. https://doi.org/10.1080/08886504.1990.10781945
5. Burke, M. J., Normand, J., & Raju, N. S. (1987). Examinee attitudes toward computer-administered ability testing. Computers in Human Behavior, 3(2), 95-107. https://doi.org/10.1016/0747-5632(87)90015-X
6. Çakır, H. (2006). Effectiveness of web and computer assisted instructions developed concerning dominant intelligence type on traffic education. (Unpublished doctoral dissertation). Ankara University, Graduate School of Educational Science, Ankara.
7. Davey, T. (2011). A guide to computer adaptive testing systems educational testing service for technical ıssues in large-scale assessment (TILSA). State Collaborative on Assessment and Student Standards (SCASS). Retrieved May 11, 2018, from https://files.eric.ed.gov/fulltext/ED543317.pdf
8. DeMars, C. (2010). Item response theory. Oxford: Oxford University Press.

9. Economides, A. A. & Roupas, C. (2007). Evaluation of computer adaptive testing systems. International Journal of Web-Based Learning and Teaching Technologies, 2(1), 70–87. https://doi.org/10.4018/jwltt.2007010104
10. Elander, J., West, R., & French, D. (1993). Behavioral correlates of individual differences in road-traffic crash risk: An examination of methodology and findings. Psychological Bulletin, 113(2), 279 294. Retrieved from https://www.researchgate.net/profile/Davina_French/publication/14748646_Behavioral_correlates_of_individual_differences_in_road traffic_crash_risk_An_examination_of_methods_and_findings/links/00b49530310a4ec836000000/Behavioral-correlates-of-individual-differences-in-road-traffic-crash-risk-An-examination-of-methods-and-findings.pdf
11. Embretson, S. E. & Reise, S. P. (2000). Item response theory for psychologists. New Jersey: Lawrence Erlbaum Associates.
12. Eroğlu, M. G. (2013). Comparison of different test termination rules in terms of measurement precision and test length in computerized adaptive testing. (Published doctoral dissertation). Hacettepe University, Graduate School of Educational Sciences, Ankara.
13. Georgiadou, E, Triantafillou, E., & Economides, A. A. (2006). Evaluation parameters for computer‐adaptive testing. British Journal of Educational Technology, 37(2), 261-278. https://doi.org/10.1111/j.1467-8535.2005.00525.x
14. Georgiadou, E., Triantafillou, E., & Economides, A. (2007). A review of item exposure control strategies for computerized adaptive testing developed from 1983 to 2005. Journal of Technology, Learning, and Assessment, 5(8), 1 39. Retrieved from https://files.eric.ed.gov/fulltext/EJ838610.pdf
15. Gökçe, S. (2012). Comparison of linear and adaptive versions of the Turkish pupil monitoring system (PMS) mathematics assessment. (Unpublished doctoral dissertation). Middle East Technical University, The Graduate School of Natural and Applied Sciences, Ankara.
16. Gülecen, M. (1998). The effect of driving courses in traffic education. (Unpublished master thesis). Gazi University, Graduate School of Natural and Applied Sciences, Ankara.
17. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Baston, MA: Kluwer Nijhoff Publishing.
18. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. USA: Sage Publications.
19. Horswill, M. S., & Coster, M. E. (2001). User-controlled photographic animations, photograph-based questions, and questionnaires: Three internet-based instruments for measuring drivers’ risk-taking behavior. Behavior Research Methods, Instruments, & Computers, 33(1), 46-58. https://doi.org/10.3758/BF03195346
20. İnal, K. (2001). Traffic safety and assessment of driving education in Turkey. (Unpublished master thesis). Gazi University, Graduate School of Natural and Applied Sciences, Ankara.
21. İşeri, A. I. (2002). Assessment of students’ mathematics achievement through computer adaptive testing procedures. (Unpublished doctoral dissertation). Middle East Technical University, The Graduate School of Natural and Applied Sciences, Ankara.
22. Kaçmaz-Omak, S. (2012). Awareness levels of primary school 5th-grade students about traffic information and traffic accidents. (Unpublished master thesis). Istanbul University, Institute of Health Sciences, Istanbul.
23. Kalender, İ. (2011). Effects of different computerized adaptive testing strategies on recovery of ability. (Unpublished doctoral dissertation). Middle East Technical University, The Graduate School of Natural and Applied Sciences, Ankara.
24. Kaptan, F. (1993). Comparison of adaptive (individualised) test application and traditional paper-pencil test application in estimation of ability. (Unpublished doctoral dissertation). Hacettepe University, Graduate School of Social Sciences, Ankara.
25. Kernan, M. C., & Howard, G. S. (1990). Computer anxiety and computer attitudes: An investigation of construct and predictive validity issues. Educational and psychological measurement, 50(3), 681-690. https://doi.org/10.1177/0013164490503026
26. Kezer, F. (2013). Comparison of computerized adaptive testing. (Published doctoral dissertation). Ankara University, Graduate School of Educational Science, Ankara.
27. Kezer, F. & Koç, N. (2014). A comparison of computerized adaptive testing. Journal of Educational Sciences Research, 4(1), 145–174.
28. Kim, J., & McLean, J. E. (1995, April). The influence of examinee test taking motivation in computerized adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
29. Kimura, T. (2017). The impacts of computer adaptive testing from a variety of perspectives. Journal of Educational Evaluation for Health Professions, 14(12), 1-5. https://doi.org/10.3352/jeehp.2017.14.12
30. Köklü, N. (1990). A comparison of tailored test, which is developed according to classical test theory, and group test. (Unpublished doctoral dissertation), Hacettepe University, Graduate School of Social Sciences, Ankara.
31. Kuyumcu, M. (2001) The ideas of the personnel and instructors who participate in the in-seervices training activities related to the traffic arranged by Gendarme Schools Command. (Unpublished master thesis). Hacettepe University, Graduate School of Social Sciences, Ankara.
32. Linacre, J. M. (2000). Computer-adaptive testing: A methodology whose time has come [MESA Memorandum No. 69]. Retrieved from https://www.cehd.umn.edu/EdPsych/C-Bas-R/Docs/Linacre2000_CAT.pdf
33. Linacre, J. M. (2006). Computer adaptive tests (CAT): Standard errors and stopping rules. Rasch Measurement Transactions, 20(2), 1062. Retrieved from www.rasch.org/rmt/rmt202f.htm
34. Linden, W. & Glas, G. (2002). Computerized adaptive testing: Theory and practice. New York: Kluver Academic Pub.
35. McKenna, F. P., & Horswill, M. S. (1999). Hazard perception and its relevance for driver licensing. Journal of the International Association of Traffic and Safety Sciences, 23(1), 36 41. Retrieved from https://trid.trb.org/Results?txtKeywords=McKenna&txtTitle=&txtSerial=%22IATSS%20Research%22&ddlSubject=&txtReportNum=&ddlTrisfile=&txtIndex=&specificTerms=&txtAgency=&txtAuthor=&ddlResultType=&chkFulltextOnly=&recordLanguage=&subjectLogic=or&dateStart=&dateEnd=&rangeType=emptyrange&sortBy=publisheddate&sortOrder=DESC&rpp=25#/View/492227
36. Ministry of Transport and Communication. (2015). Driving in Finland: Licences, driving schools, rules & vehicles. Retrieved January 21, 2017, from www.expat-finland.com/living_in_finland/driving.html
37. Miser, R. (1999). Halk eğitimi ve toplum kalkınması (Public education and community development). Ankara: MEB-Türk Tarih Kurumu Pub.
38. Mulkat-Ajibola, Y. (2015). Impact assessment of road traffic accidents on Nigerian economy. Journal of Research in Humanities and Social Science, 3(12), 8-16. Retrieved from http://eprints.federalpolyilaro.edu.ng/615/1/YUSUFFM.A12.pdf
39. Nydick, S. & Weiss, D. (2009). A hybrid simulation procedure for the development of CATs. In D. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing. Retrieved May 15, 2018, from www.psych.umn.edu/psylabs/CATCentral
40. Orcutt, V. L. (2002, February). Computerized adaptive testing: Some issues in development. Paper presented at the annual meeting of the Educational Research Exchange, University of North Texas, Denton, Texas.
41. Özbaşı, D. & Demirtaşlı, N. (2015). Development of computer literacy test as computerized adaptive testing. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 218–237. https://doi.org/10.21031/epod.79491
42. Öztuna D. (2012). A computerized adaptive testing software (CAT): SmartCAT. European Rasch Training Group (ERTG) Meeting, 17-19 April 2012, Leeds, UK.
43. Powers, D. E., & O’Neill, K. (1993). Inexperienced and anxious computer users: Coping with a computer-administered test of academic skills. Educational Assessment, 1(2), 153-173. https://doi.org/10.1207/s15326977ea0102_4
44. Revuelta, J. & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35(4) 311–327. https://doi.org/10.1111/j.1745-3984.1998.tb00541.x
45. Rudner, L. M. (1998). An online, interactive, computer adaptive testing tutorial. 11/98. Retrieved April 24, 2017, from http://EdRes.org/scripts/cat
46. Schmidt, F. L., Urry, V. W., & Gugel, J. F. (1978). Computer assisted tailored testing: Examinee reactions and evaluations. Educational and Psychological Measurement, 38(2), 265–273. https://doi.org/10.1177/001316447803800208
47. Sutton, R. E. (1991). Equity and computers in the schools: A decade of research. Review of Educational Research, 61(4), 475–503. https://doi.org/10.3102/00346543061004475
48. Tanrıkulu, S. (2002). Trafik kazalarının önlenmesi bağlamında trafik güvenliği eğitiminin rolü ve trafik kültürü. (The role of traffic safety education in the prevention of traffic accidents, and traffic culture). Polis Bilimleri Dergisi, 5(1), 45-60.
49. Thissen, D. & Mislevy, R. J. (2000). Testing algorithms. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 103-134). Mahwah (NJ): Lawrence Erlbaum.
50. Thompson, N., & Weiss, D. J. (2011). A framework for the development of computerized adaptive tests. Practical Assessment, Research & Evaluation, 16(1), 1–9. Retrieved from https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1242&context=pare
51. Turkish Statistical Institute [Türkiye İstatistik Kurumu (TÜİK)]. (2018). Highway traffic accident statistics. Retrieved November 27, 2018, from http://www.tuik.gov.tr/PreTablo.do?alt_id=1051
52. Türkoğlu, M. (2002). The Effect of the training in the private driving schools on traffic accidents. (Unpublished master thesis). Gazi University, Graduate School of Natural and Applied Sciences, Ankara.
53. Tütüncü, M. (2001). The Importance of traffic education in formal education for Turkey. (Unpublished master thesis). Gazi University, Graduate School of Natural and Applied Sciences, Ankara.
54. Vispoel, W. P. (1998). Reviewing and changing answers on computer‐adaptive and self‐adaptive vocabulary tests. Journal of Educational Measurement, 35(4), 328-345. https://doi.org/10.1111/j.1745-3984.1998.tb00542.x
55. Vispoel, W. P. (2000). Reviewing and changing answers on computerized fixed-item vocabulary tests. Educational and Psychological Measurement, 60(3), 371-384. https://doi.org/10.1177/00131640021970600
56. Vursavaş, F. (2004). Evaluation of the driving school curriculum. (Unpublished master thesis). Ankara University, Graduate School of Educational Science, Ankara.
57. Wainer, H. & Mislevy, R. J. (1990). Item response theory, item calibration and proficiency estimation. In H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, R. J. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (pp. 1-21). Hillsdale, NJ: Lawrence Erlbaum Associates.
58. Weiss, D. J. (1982). Improving measurement quality and efficincy with adaptive testing. Applied Psychological Measurement, 6(4), 473 492. https://doi.org/10.1177/014662168200600408
59. Weiss, D. J. (2004). Computerized adaptive testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37(2), 70–84. https://doi.org/10.1080/07481756.2004.11909751
60. Weiss, D. J. & Guyer, R. (2012). Manual for CATSim: Comprehensive simulation of computerized adaptive testing. St. Paul MN: Assessment Systems Corporation.
61. Weiss, D. J. & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
62. Wilder, G., Mackie, D., & Cooper, J. (1985). Gender and computers: Two surveys of computer-related attitudes. Sex Roles, 13(3), 215 228. Retrieved from https://link.springer.com/content/pdf/10.1007/BF00287912.pdf
63. Yaşar, M. (1999). A study about individualized tests [Bireyselleştirilmiş testler üzerine bir çalışma]. (Unpublished doctoral dissertation). Hacettepe University, Ankara, Turkey.
64. Young, K., Regan, M., & Hammer, M. (2007). Driver distraction: A review of the literature. In: I. J. Faulks, M. Regan, M. Stevenson, J. Brown, A. Porter & J. D. Irwin (Eds.), Distracted driving (pp. 379-405). Sydney, NSW: Australasian College of Road Safety.