Instrument Bias
A final type of method bias is called instrument bias but it does not have anything to do with the instrument, survey or test but rather refers to the experience and familiarity of the participant with test taking. There are two main types of instrument bias discussed in cross-cultural research (He, 2012), familiarity with the type of test (e.g., cognitive versus educational) and familiarity with response methods (e.g., multiple choice or rating scales).
Demetriou and colleagues describe an example of familiarity with test type (2005) when they compared Chinese and Greek children on visual-spatial tasks. The researchers found that Chinese children outperformed Greek children on the task but not because of cultural differences in visual spatial performance but because writing Chinese is a visual spatial task. Chinese children performed better because learning to write (in all cultures) requires practice and writing in Chinese language is a highly visual spatial task.
An example of how instrument bias can be reduced comes from a study that included Zambian and British children (Serpell, 1979). The children were asked to reproduce a pattern using several different types of response method including paper-and-pencil, plasticine, configurations of hand positions, and iron wire. The British children scored significantly higher on the paper-and-pencil method while the Zambians scored higher when iron wires were utilized (Serpell, 1979). These results make sense within cultural contexts. Paper pencil testing is a common experience in formal, Western education systems and making models with iron wire was a popular pastime among Zambian children. By using different response methods (i.e., paper/pencil, iron wire) the researchers were able to separate performance from bias related to response methods.
Another issue related to instrument bias is response bias, which is the systematic tendency to respond in a certain way to items or questions. There are many things that may lead to response bias including how survey questions are phrased, the demeanor of the researcher, or the desire of the participant to be a good participant and provide “the right’ answers. There are three common types of response bias:
Socially desirable responding (SDR) is the tendency to respond in a way that makes you look good. Studies that examine sensitive topics (e.g., sexuality, sexual behaviors, and mental health) or behaviors that violate social norms (e.g., fetishes, binge drinking, smoking and drug use) are particularly susceptible to SDR.
Acquiescence bias is the tendency to agree rather than disagree with items on a questionnaire. It can also mean agreeing with statements when you are unsure or in doubt. Studies have consistently shown that acquiescence response bias occurs more frequently among participants from low socioeconomic status and from collectivistic cultures (Harzing, 2006; Smith & Fischer, 2008). Additionally, work by Ross and Mirowsky (1984) found that Mexicans were more likely to engage in acquiescence and socially desirable responding than European Americans on a survey about mental health.
Extreme response bias is the tendency to use the ends of the scale (all high or all low values) regardless of what the item is asking or measuring. A demonstration of extreme response bias can be found in the work of Hui and Triandis (1989). These authors found that Hispanics tended to choose extremes on a five-point rating scale more often than did European Americans although no significant cross-cultural differences were found for 10-point scales.
