Establishing survey validity: A practical guide

Abstract

What follows is a practical guide for establishing the validity of a survey for research purposes. The motivation for providing this guide is our observation that researchers, not necessarily being survey researchers per se, but wanting to use a survey method, lack a concise resource on validity. There is far more to know about surveys and survey construction than what this guide provides; and this guide should only be used as a starting point. However, for the needs of many researchers, this guide provides sufficient, basic information on survey validity. The guide, furthermore, includes references to important handbooks for researchers needing further information.

Keywords

• Pretesting
• Cognitive interviews
• Reliability
• Research Methods
• Validity
• Survey Methods

References

1. American Education Research Association, American Psychological Association, and National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Education Research Association.
2. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. Cobern, W. W. (2000). Everyday thoughts about nature: An interpretive study of 16 ninth graders' conceptualizations of nature. Dordrecht, Netherlands: Kluwer Academic Publishers.
4. Cobern, W. W., & Adams, B. A. (2020). When interviewing: how many is enough? International Journal of Assessment Tools in Education, 7(1), 73 79. https://doi.org/10.21449/ijate.693217
5. Cobern, W. W., Adams, B. A. J., Pleasants, B. A.-S., Bentley, A., & Kagumba, R. E. (2019, March 31-April 3, 2019). Investigating the potential for unanticipated consequences of teaching the tentative nature of science. Paper presented at the National Association for Research in Science Teaching, Baltimore, MD.
6. Cobern, W. W., Gibson, A. T., & Underwood, S. A. (1999). Conceptualizations of nature: An interpretive study of 16 ninth graders' everyday thinking. Journal of Research in Science Teaching, 36(5), 541-564.
7. Cobern, W. W., Schuster, D. G., Adams, B., Skjold, B., Mugaloglu, E. Z., Bentz, A., & Sparks, K. (2014). Pedagogy of Science Teaching Tests: Formative Assessments of Science Teaching Orientations. International Journal of Science Education, 36(13), 2265-2288. Retrieved from http://bit.ly/RE95xZ
8. Baker, F. B. (2001). The basics of item response theory: ERIC Clearinghouse on Assessment and Evaluation. Retrieved from http://echo.edres.org:8080/irt/baker/final.pdf

9. Bhat, A. (2019). Top 7 Demographic survey questions for questionnaire. Retrieved from https://www.questionpro.com/blog/demographic-survey-questions/
10. Blair, J., & Conrad, F. G. (2011). Sample size for cognitive interview pretesting. Public Opinion Quarterly, 75(4), 636-658. https://doi.org/10.1093/poq/nfr035
11. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best Practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149-149. https://doi.org/10.3389/fpubh.2018.00149
12. Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how? CBE Life Sciences Education, 15(4), rm4. https://doi.org/10.1187/cbe.16-04-0148
13. Bruck, A. D., & Towns, M. (2013). Development, implementation, and analysis of a national survey of faculty goals for undergraduate chemistry laboratory. Journal of Chemical Education, 90(6), 685-693. https://doi.org/10.1021/ed300371n
14. Bruck, L. B., Towns, M., & Bretz, S. L. (2010). Faculty perspectives of undergraduate chemistry laboratory: goals and obstacles to success. Journal of Chemical Education, 87(12), 1416-1424. https://doi.org/10.1021/ed900002d
15. Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: the tailored design method (4th Ed). Hoboken, NJ: John Wiley & Sons, Inc.
16. Fryrear, A. (2016). How to write better demographic survey questions. Retrieved from https://www.surveygizmo.com/resources/blog/how-to-write-better-demographic-questions/
17. Greene, B. A. (2015). Measuring cognitive engagement with self-report scales: reflections from over 20 years of research. Educational Psychologist, 50(1), 14-30. https://doi.org/10.1080 /00461520.2014.989230
18. Haryani, E., Cobern, W. W., & Pleasants, B. A.-S. (2019). Indonesia Vocational High School Science Teachers’ Priority Regarding 21st Century Learning Skills in Their Science Classrooms. Journal of Research in Science Mathematics and Technology Education, 2(2), 105-133.
19. Krosnick, J. A. (2018). Improving question design to maximize reliability and validity. In D. L. Vannette & J. A. Krosnick (Eds.), The Palgrave handbook of survey research (pp. 95-101). New Ydork: Palgrave Macmillan.
20. Lamb, R. L., Annetta, L., Meldrum, J., & Vallett, D. (2012). Measuring science interest: RASCH validation of the science interest survey. International Journal of Science and Mathematics Education, 10(3), 643-668. https://doi.org/10.1007/s10763-011-9314-z
21. Luo, T., Wang, J., Liu, X., & Zhou, J. (2019). Development and application of a scale to measure students’ STEM continuing motivation. International Journal of Science Education, 41(14), 1885-1904. https://doi.org/10.1080/09500693.2019.1647472
22. Multon, K. D. (2010). Test-retest reliability. In N. J. Salkind (Ed.), Encyclopedia of research design (pp. 1495-1498). Thousand Oaks, California: SAGE Publications, Inc.
23. Nyutu, E. N., Cobern, W. W., & Pleasants, B. A.-S. (2020). Development of an instrument to assess students’ perceptions of their undergraduate laboratory environment. The Journal for Research and Practice in College Teaching, 5(1), 1-18. Retrieved from https://journals.uc.edu/index.php/jrpct/article/view/1492
24. Redline, C. (2013). Clarifying categorical concepts in a web survey. Public Opinion Quarterly, 77(S1), 89–105. https://doi.org/10.1093/poq/nfs067
25. Rosenberg, S. (2017). Respectful collection of demographic data. Retrieved from https://medium.com/@anna.sarai.rosenberg/respectful-collection-of-demographic-data-56de9fcb80e2
26. Ruel, E. E., Wagner III, W. E., & Gillespie, B. J. (2016). The practice of survey research: theory and applications. Los Angeles, CA: SAGE.
27. Seidman, I. E. (2006). Interviewing as qualitative research: a guide for researchers in education and the social sciences, 3rd Edition. New York: Teachers College, Columbia University.
28. Smyth, J. D. (2016). Chapter 16: Designing Questions and Questionnaires. In C. Wolf, D. Joye, T. W. Smith, & Y.-c. Fu (Eds.), The SAGE handbook of survey methodology (pp. 218-235). https://doi.org/10.4135/9781473957893
29. Staus, N. L., Lesseig, K., Lamb, R., Falk, J., & Dierking, L. (2019). Validation of a measure of STEM interest for adolescents. International Journal of Science and Mathematics Education. https://doi.org/10.1007/s10763-019-09970-7
30. Taber, K. S. (2018). The use of Cronbach's Alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273-1296. Retrieved from https://doi.org/10.1007/s11165-016-9602-2
31. Vannette, D. L., & Krosnick, J. A. (2018). The Palgrave handbook of survey research. New York: Palgrave Macmillan.
32. Willis, G. B. (2016). Chapter 24: Questionnaire pretesting. In C. Wolf, D. Joye, T. W. Smith, & Y.-c. Fu (Eds.), The SAGE handbook of survey methodology (pp. 359-380). https://doi.org/10.4135/9781473957893
33. Willis, G. B. (2018). Cognitive interviewing in survey design: State of the science and future directions. In D. L. Vannette & J. A. Krosnick (Eds.), The Palgrave handbook of survey research (pp. 103-107). New York: Palgrave Macmillan.