Principles for Minimizing Errors in Examination Papers and Other Educational Assessment Instruments

Abstract

Errors in examination papers and other assessment instruments can compromise fairness. For example, a history question containing an incorrect historical date could be impossible for students to answer. Incorrect instructions at the start of an examination could lead students to answer the wrong number of questions. As there is little research on this issue within the educational assessment community, we reviewed the literature on minimizing errors in other industries and domains, including aviation, energy, and medicine. We identified generalizable principles and applied them to our context of educational assessment. We argue that since assessment instrument construction is a complex system comprising numerous interacting components, a holistic approach to system improvement is required. Assessment instrument errors stem primarily from human failure. When human failure occurs, it is not good enough to suggest that ‘to err is simply human’. Instead it is necessary to look deeper, evaluating the latent working conditions that underpin the efficacy of procedures, making the human failure more or less likely. Drawing from the aviation industry’s ergonomic SHELLO model, we articulate and explore three of the most critical working conditions that relate to our context: (i) time pressure, (ii) workload and stress, and (iii) wider organizational culture, including good error data collection. We conclude with recommendations for best practice in minimizing errors in assessment instruments. A ‘good’ error culture should be promoted, which avoids blaming individuals. Errors should be acknowledged readily by all, and system owners should take a scientific approach to understanding and learning from them.

Keywords

• Error
• Fairness
• Instrument design
• Examination design

References

1. Achtenhagen, F. (1994, June). Presentation to Third International Conference of Learning at Work, Milan, Italy.
2. Akinci, B. (2014). Situational Awareness in Construction and Facility Management. Frontiers of Engineering Management, 1(3), 283-289. https://doi.org/10.15302/J-FEM-2014037
3. Aimola Davies, A., Waterman, S., White, R. & Davies, M. (2013). When you fail to see what you were told to look for: Inattentional blindness and task instructions. Consciousness and Cognition, 22(1), 221-230. https://doi.org/10.1016/j.concog.2012.11.015
4. Baddeley, A. (2010). Working memory. Current Biology, 20(4), R136-R140.
5. Baranowski, R. (2006). Item editing and editorial review. In S. Downing & T. Haladyna (Eds.), Handbook of Test Development (pp. 349-357). Lawrence Erlbaum Associates.
6. Battmann, W., & Klumb, P. (1993). Behavioural economics and compliance with safety regulations. Safety Science, 16, 35 46. https://www.sciencedirect.com/science/article/pii/092575359390005X BBC (2017, November 21) NZ minister orders probe into 'impossible' maths exam. https://www.bbc.co.uk/news/blogs-news-from-elsewhere-42065574
7. Bleetman, A., Sanusi, S., Dale, T., & Brace, S. (2012). Human factors and error prevention in emergency medicine. Emergency Medicine Journal, 29, 389 393. http://emj.bmj.com/content/29/5/389.long
8. Borchard, E. M. (2013). Convicting the innocent and state indemnity for errors of criminal justice. The Justice Institute. (Original work published 1932)

9. Bruner, J. S., & Postman, L. (1949). On the perception of incongruity: a paradigm. Journal of Personality, 18(2), 206 23. https://doi.org/10.1111/j.1467 6494.1949.tb01241.x
10. Chang, Y.-H., & Wang, Y.-C. (2010). Significant human risk factors in aircraft maintenance technicians. Safety Science, 48(1), 54 62. https://doi.org/10.1016/j.ssci.2009.05.004
11. Chang, Y.-H., Yang, H.-H., & Hsiao, Y.-J. (2016). Human risk factors associated with pilots in runway excursions. Accident Analysis & Prevention 94, 227-237. https://doi.org/10.1016/j.aap.2016.06.007
12. Civil Aviation Authority (2014). Flight-crew human factors handbook. CAA. http://publicapps.caa.co.uk/docs/33/CAP%20737%20DEC16.pdf
13. Dekker, S. (2002a). The Field Guide to Human Error Investigations. Ashgate.
14. Dekker, S. (2002b). Reconstructing human contributions to accidents: the new view on error and performance. Journal of Safety Research, 33(3), 371-385. https://doi.org/10.1016/S0022-4375(02)00032-4
15. Dekker, S. (2017) Just Culture: Restoring trust and accountability in your organization. CRC Press.
16. Edmundson, A. (1999). Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly, 44(2), 350-383.
17. Edward, E. (1972). Man and machine: systems for safety. Proceedings of the British Airline Pilots Association Technical Symposium, London.
18. Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64.
19. Festinger, L. (1957). A Theory of Cognitive Dissonance. Stanford University Press.
20. Flin, R., O’Connor, P., & Mearns, K. (2002). Crew resource management: improving team work in high reliability industries. Team Performance Management: An International Journal, 8(3/4), 68-78. https://doi.org/10.1108/13527590210433366
21. Förster, J., Higgins, E. T., & Bianco, A. T. (2003). Speed/accuracy decisions in task performance: Built-in trade-off or separate strategic concerns? Organizational Behavior and Human Decision Processes, 90(1), 148 164. https://www.sciencedirect.com/science/article/pii/S0749597802005095
22. Gawande, A. (2011). The checklist manifesto: How to get things right. Metropolitan Books.
23. Harrison, A. (2011, June 9) Students hit by more exam errors. BBC. https://www.bbc.co.uk/news/education-13710868
24. Higgins, E. T. (1997). Beyond pleasure and pain. American Psychologist, 52(12), 1280-1300. https://pdfs.semanticscholar.org/6b64/5e0418ae70e82cc322dd6fbf0647ae2523e4.pdf
25. Innocence Project (2021). The innocence project. https://www.innocenceproject.org
26. International Civil Aviation Organization (1993). Investigation of Human Factors in Accidents and Incidents. Human Factor Digest No.7. https://skybrary.aero/bookshelf/books/2037.pdf
27. Jones, D. G., & Endsley, M. R. (1996). Sources of situation awareness errors in aviation. Aviation, Space, and Environmental Medicine, 67(6), 507-512.
28. Kahneman, D. (2011). Thinking, fast and slow. Penguin Group.
29. Kontogiannis, T., & Malakis, S. (2009). A proactive approach to human error detection and identification in aviation and air traffic control. Safety Science, 47, 693-706.
30. Meredith, R. (2019, May 17) AS-level Economics exam error under investigation in NI. BBC. https://www.bbc.co.uk/news/uk-northern-ireland-48313904
31. Mitleton-Kelly, E. (2003). Ten principles of complexity and enabling infrastructures. In E. Mitleton-Kelly (Ed.), Complex systems and evolutionary perspectives on organisations: the application of complexity theory to organisations. Elsevier.
32. New Straits Times (2015) S. Korea exam chief resigns over errors in high-stakes college test. https://www.nst.com.my/news/2015/09/s-korea-exam-chief-resignsover-errors-high-stakes-college-test
33. Nisbet, I., & Shaw, S. (2020). Is Assessment Fair? Sage.
34. Oates, T. (2017). A Cambridge Approach to improving education. Cambridge Assessment. http://www.cambridgeassessment.org.uk/Images/cambridge approach to improving education.pdf
35. Ofqual (2019). GCSE, AS & A level summer report 2018. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/852440/GQ-Summer-Report-2019-MON1100.pdf
36. Popper, K. (1963). Conjectures and refutations: The growth of scientific knowledge. Routledge and Kegan Paul.
37. Pronovost, P., & Vohr, E. (2011). Safe Patients, Smart Hospitals: How One Doctor's Checklist Can Help Us Change Health Care from the Inside Out. Penguin books.
38. Reason, J. (1990). Human error. Cambridge University Press.
39. Reason, J. (2008). The human contribution. Ashgate.
40. Reason, J. (2013). A life in error: from little slips to big disasters. Ashgate.
41. Richardson, H. (2017, May 26) GCSE exam error: Board accidentally rewrites Shakespeare. BBC. https://www.bbc.co.uk/news/education-40059967
42. Rhoades, K., & Madaus, G. (2003). Errors in standardized tests: A systemic problem. Boston College.
43. Rodriguez, M. (2015). Selected-response item development. In S. Lane, M. Raymond, & T. Haladyna (Eds.), Handbook of Test Development (pp. 259-273). Routledge.
44. Syed, M. (2015). Black box thinking. Marginal gains and the secrets of high performance. John Murray.
45. Verne, J. (1996). Journey to the centre of the earth. Wordsworth Editions Limited. (Original work published 1864.)
46. Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (1999). Organising for high reliability: Processes of collective mindfulness. In R. S. Sutton & B. M. Staw (Eds.). Research In Organizational Behavior, 21, 23-81.
47. Wickens, C. D. (2008). Situation awareness: Review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Human Factors, 50(3), 397 403. https://journals.sagepub.com/doi/pdf/10.1518/001872008X288420
48. Wiegmann, D. A., & Shappell, S. A. (2003). A Human Error Approach to Aviation Accident Analysis. Ashgate.