Automated Quality Assurance of Educational Testing

Year 2018, Volume: 19 Issue: 3, 71 - 92, 18.07.2018

Rositsa Doneva Silvia Gaftandzhıeva George Totkov

https://doi.org/10.17718/tojde.444961

Cited By: 1

Abstract

This paper presents a study on known approaches for quality assurance of educational test and test items. On its basis a comprehensive approach to the quality assurance of online educational testing is proposed to address the needs of all stakeholders (authors of online tests, teachers, students, experts, quality managers, etc.). According to the proposed approach is developed an original software application Test Quality Evaluation (TQE) for the automation of the stakeholders’ activities for quality assurance of educational tests throughout the whole lifecycle. The application retrieves and provides analysis of data from online tests conducted and specially designed surveys for quality evaluation of educational tests by students and experts. It allows tracking and evaluating the quality of educational tests in real time and provides the related quantitative data in different levels of generalization – in the level of a separate educational test, of educational tests of an entire course, or educational tests of a subject area. The software application has been put under real-time testing for quality evaluation of educational tests, included in e-learning courses from different subject areas that prove its applicability.

Keywords

Assessment quality, educational testing, automated quality assurance, online tests and test items quality

References

• Amouei, A., Barari, R., Naghipour D., Mortazavi Y., & Hosseini S. Reza (2014). Evaluation of Multiple Choice Questions Quality Trend as Structure and Taxonomy. FUTURE of MEDICAL EDUCATION JOURNAL, 4(3), 26-30. APA (2014). The Standards for Educational and Psychological Testing, Retrieved July 2, 2017, from http://www.apa.org/science/programs/testing/standards aspx#overview Blackboard Help (2017). Running Item Analysis on a Test, Retrieved July 2, 2017, from https://en-us.help.blackboard.com CDC (2017). Checklist to Evaluate the Quality of Questions, Retrieved April 12, 2017, from http://www.cdc.gov/healthyyouth/ evaluation/ index.htm CFATIQC (2017). Common Formative Assessment Test Item Quality Checklist, Retrieved July 2, 2017, from https://docs.google.com/document/preview?hgd=1&id= 1vmkf0UAU21u8bGRRwtFb__4kw6NThEDM8WD9NyMgSFU&pli=1 CITL. (2017). Improving Your Test Questions. Retrieved July 2, 2017, from http://cte.illinois.edu/testing/exam/test_ques3.html 91 Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (ed.). Educational Measurement, 2nd ed., 443–507. Washington, D.C.: American Council on Education. Dill, D. D. (2010). “Quality Assurance in Higher Education: Practices and Issues.” In P. P. Peterson, E. Baker, and B. McGaw, (eds.). International Encyclopedia of Education. Third Edition. 377-383. Doneva, R., & Gaftandzhieva, S. (2015). Automated e-learning quality evaluation. Proceedings of the International Conference on e-Learning. Berlin, Germany. ISSN 2376-6698, 156-162. EFPA (2013). Revised EFPA Review Model for the Description and Evaluation of Psychological and Educational Tests, Test Review Model Version 4.2.6, Retrieved July 2, 2017, from http://www.efpa.eu/download/650d0d4ecd407a51139ca44ee704fda4 EUSHARE (2015). European Standards and Guidelines for Quality Assurance in the EHEA. Belgium: EUSHARE. Gaftandzhieva S. (2016). Automated Evaluation of Students’ Satisfaction, International Journal of Information Technologies and Security (IJITS), 8(1), 31-40. Gaftandzhieva S. (2017). A Model and System for Dynamic Quality Evaluation in Higher Education. (Doctoral dissertation). Available from University of Plovdiv. Gierl, M.J., & Hollis, L. (2013). Evaluating the quality of medical multiple-choice items created with automated processes, Medical Education 2013, 47(7), 726–733, doi: 10.1111/medu.12202. Hambleton, RK, Swaminathan, H, Rogers, HJ (1991). Fundamentals of items response theory. Newbury Park (California): Sage Publications. 174 p. Hamilton, L. S., Stecher, Br. M.,& Klein St.P. (2002). Making Sense of Test-Based Accountability in Education. RAND. ISO 9000:2015 (2015). Quality management systems — Fundamentals and vocabulary, Retrieved July 2, 2017, from https://www.iso.org/obp/ui/#iso:std:iso:9000:ed4:v1:en JasperSoft (2017). Business Intelligence Solutions. Retrieved July 2, 2017, from http://www.jaspersoft.com/business-intelligence-solutions Legault N. (2017). Post-Course Evaluations for е-Learning: 60+Questions to Include, Retrieved July 2, 2017, from https://community.articulate.com/articles/postcourse-evaluations-for-e-learning-60-questions-to-include Machado-da-Silva, F, Meirelles, F , Filenga, D , Filho, M . (2015). Student Satisfaction Process In Virtual Learning System: Considerations Based In Information And Service Quality From Brazil’s Experience. Turkish Online Journal of Distance Education, 15 (3), 122-142. DOI: 10.17718/tojde.52605. Mark, D. R. (1985). The Difficulty of Test Items That Measure More Than One Ability, Applied Psychological Measurement, 9(4), 401 – 412. Messick, S. V. (1989). Educational Measurement. 3rd ed., 13–103. New York: Macmillan. Moodle Documentation (2017). Quiz statistics report, Retrieved July 2, 2017, from https://docs.moodle.org/29/en/Quiz_statistics_report. Mutiara, D., Zuhairi, A., & Kurniati S. (2007). Designing, Developing, Producing And Assuring The Quality of Multi-Media Learning Materials For Distance Learners: Lessons Learnt From Indonesia's Universitas Terbuka. Turkish Online Journal of Distance Education-TOJDE, ISSN 1302–6488, 8(2), 95-112. 92 Professional Testing (2017). How do you Determine if a Test has Validity, Reliability, Fairness, and Legal Defensibility?. Retrieved July 2, 2017, from http://www.proftesting.com/test_topics/test_quality.php Pyrczak, F. (1973). Validity of the Discrimination Index as a Measure of Item Quality. Journal of Educational Measurement, 10(3), 227-231. Rasch (2017). Rasch Software, Retrieved July 2, 2017, from https://www.rasch.org/rmt/rmt114d.htm Saad, S., Carter, G.W., Rothenberg, M., & Israelson, E. (1999). Testing and Assessment: an Employer’s Guide to Good Practises, Employment and Training Administration (DOL). Washington, DC. Office of Policy and Research. Thompson, B., & Levitov, J.E. (1985). Using microcomputers to score and evaluate test items. Collegiate Microcomputer, 3, 163-168. Totkov, G., Gaftandzhieva, S., & Doneva, R. (2016). Dynamic Quality Evaluation in Higher Education (with application in e-Learning). Proceedings of the First Varna Conference on E-learning and Knowledge Management: Bridging the Gap between Secondary and Higher Education, 8-23. Totkov, G., Raikova, М., & Kostadinova, H. (2014). The test in e-learning. Plovdiv: “Rakursi” LTD.