Quality Control in Survey Design: Evaluating a Survey of Educators’ Attitudes Concerning Differentiated Compensation

Abstract

This study utilized the Rasch model to assess the quality of a survey instrument designed to measure attitudes of administrators and teachers concerning a differentiated teacher compensation program piloted in Kentucky. Researchers addressing potentially contentious issues should ensure their methods stand up to rigorous criticism. The results indicate that the rating scale does not function as expected, with items being too easy to endorse. Future iterations of this survey should be revised prior to release. Recommendations for improvement are provided.

Keywords

• Rasch measurement
• Survey design
• Rating Scale
• Teacher Compensation

Quality Control in Survey Design: Evaluating a Survey of Educators’ Attitudes Concerning Differentiated Compensation

Abstract

This study utilized the Rasch model to assess the quality of a survey instrument designed to measure attitudes of administrators and teachers concerning a differentiated teacher compensation program piloted in Kentucky. Researchers addressing potentially contentious issues should ensure their methods stand up to rigorous criticism. The results indicate that the rating scale does not function as expected, with items being too easy to endorse. Future iterations of this survey should be revised prior to release. Recommendations for improvement are provided.

Keywords

• Rasch measurement
• Survey design
• Rating Scale
• Teacher Compensation

References

1. Andrich D., (1978). A Rating Formulation for Ordered Response Categories. Psychometrika, 43(4), 561-573.
2. Becker G., (2001). Controlling Decremental and Inflationary Effects in Reliability Estimation Resulting from Violations of Assumptions. Psychological Reports, 89(2), 403-424. doi: 10.2466/pr0.2001.89.2.403
3. Bond T.G. & Fox C.M., (2001). Applying the Rasch Model: Fundamental Measurement in the Human Sciences. Mahwah, NJ: Lawrence Erlbaum.
4. Bradley K.D. & Sampson S.O., (2005). A case for using Rasch Rating Scale analysis to assess the quality of measurement in survey research. The Respondent, 12-13.
5. Fisher W., (1992). Reliability, Separation, Strata Statistics. Rasch Measurement Transactions, 6(3), 238.
6. Guttman L., (1944). A Basis for Scaling Qualitative Data. American Sociological Review, 9(2), 139-150.
7. Hambleton R.K., Swaminathan, H., & Rogers, H.J., (1991). Fundamentals of Item Response Theory. New York: Sage Publications.
8. Linacre J.M., (2002a). Optimizing Rating Scale Category Effectiveness. Journal of Applied Measurement, 3(1), 85-106.

9. Linacre J.M., (2002b). What do Infit and Outfit, Mean-square and Standardized mean? Rasch Measurement Transactions, 16(2), 878.
10. Linacre J.M., (2004). Winsteps Rasch Measurement computer program (Version 3.51). Beaverton, OR: Winsteps.com.
11. Rasch G., (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark.: Danish Institute for Educational Research.
12. Wright B.D. (1996). Reliability and Separation. Rasch Measurement Transactions, 9(4), 472.
13. Wright B.D., (1997). Fundamental Measurement for Outcome Evaluation. Physical Medicine and Rehabilitation: State of the Art Reviews, 11(2), 261-288.
14. Wright B.D. & Masters G.N., (1982). Rating scale analysis. Chicago: MESA Press.
15. Wright B.D. & Stone M.H., (1979). Best Test Design. Chicago: MESA Press.