Understand response bias: a threat to measurement validity and measurement equivalence

Rating scales are commonly used in psychological surveys to elicit respondents’ judgements. However, the presence of response bias will threat the validity of result in surveys. Response bias (RB) refer to the cases where the number of certain response options was disproportionately more than others. The causes of RB consist of both respondent factors (such as personality or cultural influence) and context-dependent factors (such as scale format or nature of contents). The results before and after controlling for RB can be completely different.This study aimed to investigate the influences of RB. A series of simulation studies was carried out to explore the influence of RB on means, variances and associations across different conditions. The influence of RB on variables was evaluated by several indicators, including bias in estimation and variance ratios. Results showed that the influence of RB depended on the shapes of distributions of the variables. In addition, we used the data from the World Value Survey (WVS) Wave 6 to demonstrate how RB could influence means, variances and associations among variables in real world. We found that RB had substantially differing impacts on the means, variances and distribution shapes of the WVS data across different countries. Taken together, the simulation results and WVS findings indicate that RB can be a major challenge for measurement validity and measurement equivalence in studies using rating scales. We discuss implications and recommendations to researchers.