A data pipeline for e-large-scale assessments: Better automation, quality assurance, and efficiency

Abstract

The increasing volume of large-scale assessment data poses a challenge for testing organizations to manage data and conduct psychometric analysis efficiently. Traditional psychometric software presents barriers, such as a lack of functionality for managing data and conducting various standard psychometric analyses efficiently. These challenges have resulted in high costs to achieve the desired research and analysis outcomes. To address these challenges, we have designed and implemented a modernized data pipeline that allows psychometricians and statisticians to efficiently manage the data, conduct psychometric analysis, generate technical reports, and perform quality assurance to validate the required outputs. This modernized pipeline has proven to scale with large databases, decrease human error by reducing manual processes, efficiently make complex workloads repeatable, ensure high quality of the outputs, and reduce overall costs of psychometric analysis of large-scale assessment data. This paper aims to provide information to support the modernization of the current psychometric analysis practices. We shared details on the workflow design and functionalities of our modernized data pipeline, which provide a universal interface to large-scale assessments. The methods for developing non-technical and user-friendly interfaces will also be discussed.

Keywords

• Data pipelines
• Psychometric analysis
• Large-scale assessments
• Data validation
• Reporting

References

1. Addey, C., & Sellar, S. (2018). Why do countries participate in PISA? Understanding the role of international large-scale assessments in global education policy. In A. Verger, H.K. Altinyelken, & M. Novelli (Eds.), Global education policy and international development: New agendas, issues and policies (3rd ed., pp. 97–117). Bloomsbury Publishing.
2. Allaire, J., Xie, Y., McPherson, J., Luraschi, J., Ushey, K., Atkins, A., ... & Iannone, R. (2022). rmarkdown: Dynamic Documents for R. R package version, 1(11).
3. Ansari, G.A., Parvez, M.T., & Al Khalifah, A. (2017). Cross-organizational information systems: A case for educational data mining. International Journal of Advanced Computer Science and Applications, 8(11), 170 175. http://dx.doi.org/10.14569/IJACSA.2017.081122
4. Azab, A. (2017, April). Enabling docker containers for high-performance and many-task computing. In 2017 ieee international conference on cloud engineering (ic2e) (pp. 279-285). IEEE.
5. Bezanson, J., Karpinski, S., Shah, V.B., & Edelman, A. (2012). Julia: A fast dynamic language for technical computing. ArXiv Preprint ArXiv:1209.5145.
6. Bertolini, R., Finch, S.J., & Nehm, R.H. (2021). Enhancing data pipelines for forecasting student performance: Integrating feature selection with cross-validation. International Journal of Educational Technology in Higher Education, 18(1), 1 23. https://doi.org/10.1186/s41239-021-00279-6
7. Bertolini, R., Finch, S.J., & Nehm, R.H. (2022). Quantifying variability in predictions of student performance: Examining the impact of bootstrap resampling in data pipelines. Computers and Education: Artificial Intelligence, 3, 100067. https://doi.org/10.1016/j.caeai.2022.100067
8. Bryant, W. (2019). Developing a strategy for using technology-enhanced items in large-scale standardized tests. Practical Assessment, Research, and Evaluation, 22(1), 1. https://doi.org/10.7275/70yb-dj34

9. Camara, W.J., & Harris, D.J. (2020). Impact of technology, digital devices, and test timing on score comparability. In M.J. Margolis, R.A. Feinberg (Eds.), Integrating timing considerations to improve testing practices (pp. 104-121). Routledge.
10. Chalmers. R.P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1 29. https://doi.org/10.18637/jss.v048.i06
11. Croudace, T., Ploubidis, G., & Abbott, R. (2005). BILOG-MG, MULTILOG, PARSCALE and TESTFACT. British Journal of Mathematical & Statistical Psychology, 58(1), 193. https://doi.org/10.1348/000711005X37529
12. Desjardins, C.D., & Bulut, O. (2018). Handbook of educational measurement and psychometrics using R. CRC Press.
13. Dogaru, I., & Dogaru, R. (2015, May). Using Python and Julia for efficient implementation of natural computing and complexity related algorithms. In 2015 20th International Conference on Control Systems and Computer Science (pp. 599-604). IEEE.
14. Dowle, M., & Srinivasan, A. (2023). data.table: Extension of 'data.frame'. https://r-datatable.com,https://Rdatatable.gitlab.io/data.table.
15. du Toit, M. (2003). IRT from SSI: BILOG-MG, MULTILOG, PARSCALE, TESTFACT. Scientific Software International.
16. Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Erlbaum.
17. Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory (Vol. 2). Sage.
18. Kamens, D.H., & McNeely, C.L. (2010). Globalization and the growth of international educational testing and national assessment. Comparative education review, 54(1), 5-25. https://doi.org/10.1086/648471
19. Goodman, D.P., & Hambleton, R.K. (2004). Student test score reports and interpretive guides: Review of current practices and suggestions for future research. Applied Measurement in Education, 17(2), 145-220. https://doi.org/10.1207/s15324818ame1702_3
20. Liu, O.L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M.C. (2014). Automated scoring of constructed‐response science items: Prospects and obstacles. Educational Measurement: Issues and Practice, 33(2), 19-28. https://doi.org/10.1111/emip.12028
21. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Addison Wesley, Reading, MA.
22. Lynch, S. (2022). Adapting paper-based tests for computer administration: Lessons learned from 30 years of mode effects studies in education. Practical Assessment, Research, and Evaluation, 27(1), 22.
23. IBM (2020). IBM SPSS Statistics for Windows, Version 27.0. IBM Corp.
24. Martinková, P., & Drabinová, A. (2018). ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. R Journal, 10(2), 503-515.
25. Merkel, D. (2014). Docker: Lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2.
26. Microsoft Corporation. (2018). Microsoft Excel. Retrieved from https://office.microsoft.com/excel
27. Moncaleano, S., & Russell, M. (2018). A historical analysis of technological advances to educational testing: A drive for efficiency and the interplay with validity. Journal of Applied Testing Technology, 19(1), 1–19.
28. Morandat, F., Hill, B., Osvald, L., & Vitek, J. (2012). Evaluating the design of the R language: Objects and functions for data analysis. In ECOOP 2012–Object-Oriented Programming: 26th European Conference, Beijing, China, June 11-16, 2012. Proceedings 26 (pp. 104-131). Springer Berlin Heidelberg.
29. Muraki, E., & Bock, R.D. (2003). PARSCALE 4 for Windows: IRT based test scoring and item analysis for graded items and rating scales [Computer software]. Scientific Software International, Inc.
30. Oranje, A., & Kolstad, A. (2019). Research on psychometric modeling, analysis, and reporting of the national assessment of educational progress. Journal of Educational and Behavioral Statistics, 44(6), 648-670. https://doi.org/10.3102/1076998619867105
31. R Core Team (2022). R: Language and environment for statistical computing. (Version 4.2.1) [Computer software]. Retrieved from https://cran.r-project.org.
32. Reise, S.P., Ainsworth, A.T., & Haviland, M.G. (2005). Item response theory: Fundamentals, applications, and promise in psychological research. Current directions in psychological science, 14(2), 95-101.
33. Rupp, A.A. (2003). Item response modeling with BILOG-MG and MULTILOG for Windows. International Journal of Testing, 3(4), 365 384. https://doi.org/10.1207/S15327574IJT0304_5
34. Russell, M. (2016). A framework for examining the utility of technology-enhanced items. Journal of Applied Testing Technology, 17(1), 20-32.
35. Rutkowski, L., Gonzalez, E., Joncas, M., & Von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142-151. https://doi.org/10.3102/0013189X10363170
36. Scalise, K., & Gifford, B. (2006). Computer-based assessment in e-learning: A framework for constructing" intermediate constraint" questions and tasks for technology platforms. The Journal of Technology, Learning and Assessment, 4(6).
37. Schauberger, P., & Walker, A (2022). openxlsx: Read, Write and Edit xlsx Files. https://ycphs.github.io/openxlsx/index.html, https://github.com/ycphs/openxlsx
38. Schleiss, J., Günther, K., & Stober, S. (2022). Protecting student data in ML Pipelines: An overview of privacy-preserving ML. In International Conference on Artificial Intelligence in Education (pp. 532-536). Springer, Cham.
39. Schloerke, B., & Allen, J. (2023). plumber: An API Generator for R. https://www.rplumber.io, https://github.com/rstudio/plumber
40. Schumacker, R. (2019). Psychometric packages in R. Measurement: Interdisciplinary Research and Perspectives, 17(2), 106-112. https://doi.org/10.1080/15366367.2018.1544434
41. Skiena, S.S. (2017). The data science design manual. Springer.
42. Sung, K.H., Noh, E.H., & Chon, K.H. (2017). Multivariate generalizability analysis of automated scoring for short answer items of social studies in large-scale assessment. Asia Pacific Education Review, 18, 425-437. https://doi.org/10.1007/s12564-017-9498-1
43. Thissen, D., Chen, W-H, & Bock, R.D. (2003). MULTILOG 7 for Windows: Multiple category item analysis and test scoring using item response theory [Computer software]. Scientific Software International, Inc.
44. Van Rossum, G., & Drake Jr, F.L. (1995). Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam.
45. Volante, L., & Ben Jaafar, S. (2008). Educational assessment in Canada. Assessment in Education: Principles, Policy & Practice, 15(2), 201 210. https://doi.org/10.1080/09695940802164226
46. Weber, B. G. (2020). Data science in production: Building scalable model pipelines with Python. CreateSpace Independent Publishing.
47. Wickham, H. (2022). stringr: Simple, consistent wrappers for common string operations. https://stringr.tidyverse.org.
48. Wickham, H., François, R., Henry, L., & Müller, K. (2022). dplyr: A grammar of data manipulation. Retrieved from https://dplyr.tidyverse.org.
49. Wickham, H., & Girlich, M. (2022). tidyr: Tidy messy data. Retrieved from https://tidyr.tidyverse.org
50. Wise, S.L. (2018). Computer-based testing. In the SAGE Encyclopedia of Educational Research, Measurement, and Evaluation (pp. 341–344). SAGE Publications, Inc.
51. Ysseldyke, J., & Nelson, J.R. (2002). Reporting results of student performance on large-scale assessments. In G. Tindal & T.M. Haladyna (Eds.), Large-scale assessment programs for all students: Validity, technical adequacy, and implementation. (pp. 467-483). Routledge
52. Zenisky, A.L., & Sireci, S.G. (2002). Technological innovations in large-scale assessment. Applied Measurement in Education, 15(4), 337 362. https://doi.org/10.1207/S15324818AME1504_02