Understanding Cultural Bias in Psychometric Assessments: Item, Method, and Construct | Lecture notes Communication

12C Sources of cultural bias

A lack of equivalence can be due to a host of reasons, which are referred to with

the generic term of cultural bias. The effects of a biasing factor can be limited to

a single item or a few items (e.g., translation errors). This is called item bias or

differential item functioning (DIF). The effects can also infl uence the responses

on most or all items, leading to method bias. Finally, bias can pertain to the op-

erationalization or defi nition of a concept; this is referred to as concept bias or

construct bias. An overview of various sources of bias is presented in Additional

Topics Table 12.2. More extensive overviews can be found in Van de Vijver and

Tanzer (2004) and Van de Vijver and Leung (1997).

It is clear that there is a correspondence between levels of equivalence and

sources of bias. This is most evident at the level of concept bias: unequal notions

about concepts will lead to systematic differences in concept representation across

cultures. The consequences for equivalence are least clear in the case of item bias.

If only a single item or a few items show evidence of bias, they can be eliminated

and this improves the equivalence of scores. However, evidence of item bias can

also be taken as an indication that an instrument does not represent precisely

identical traits.

The notion of item bias is immediately clear if one thinks of an example. An

item asking for the name of the capital of a country should be much easier for na-

tionals of that country than in other countries. However, item bias can also be the

consequence of subtle differences in shades of meaning. An item is identifi ed as

biased if test takers with the same score on the instrument but belonging to differ-

ent populations do not have the same probability of giving a certain answer (e.g.,

“yes” answer, correct answer) for the item (Shepard, Camilli and Averill, 1981).

Thus, the test score variable on the instrument as a whole is used as the standard to

evaluate each separate item. Some simple bias indices have been proposed, such as

the correlation between the diffi culty levels of ability test items in two cultures, or

between the levels of endorsement of questionnaire items. Another index of item

bias is the interaction between items and culture in an analysis of variance.

Psychometrically more sophisticated techniques are based on item response

theory or loglinear models. A popular technique for analyzing bias in dichotomous

items (e.g., yes–no; right–wrong) is the Mantel–Haenszel procedure (Holland and

Wainer, 1993). For items with a graded response scale better use is made of the

9780521745208_c12C.indd 89780521745208_c12C.indd 8 1/29/11 1:01:29 PM1/29/11 1:01:29 PM

Partial preview of the text

Download Understanding Cultural Bias in Psychometric Assessments: Item, Method, and Construct and more Lecture notes Communication in PDF only on Docsity!

12C Sources of cultural bias

A lack of equivalence can be due to a host of reasons, which are referred to with the generic term of cultural bias. The effects of a biasing factor can be limited to a single item or a few items (e.g., translation errors). This is called item bias or differential item functioning (DIF). The effects can also influence the responses on most or all items, leading to method bias. Finally, bias can pertain to the op- erationalization or definition of a concept; this is referred to as concept bias or construct bias. An overview of various sources of bias is presented in Additional Topics Table 12.2. More extensive overviews can be found in Van de Vijver and Tanzer (2004) and Van de Vijver and Leung (1997). It is clear that there is a correspondence between levels of equivalence and sources of bias. This is most evident at the level of concept bias: unequal notions about concepts will lead to systematic differences in concept representation across cultures. The consequences for equivalence are least clear in the case of item bias. If only a single item or a few items show evidence of bias, they can be eliminated and this improves the equivalence of scores. However, evidence of item bias can also be taken as an indication that an instrument does not represent precisely identical traits. The notion of item bias is immediately clear if one thinks of an example. An item asking for the name of the capital of a country should be much easier for na- tionals of that country than in other countries. However, item bias can also be the consequence of subtle differences in shades of meaning. An item is identified as biased if test takers with the same score on the instrument but belonging to differ- ent populations do not have the same probability of giving a certain answer (e.g., “yes” answer, correct answer) for the item (Shepard, Camilli and Averill, 1981). Thus, the test score variable on the instrument as a whole is used as the standard to evaluate each separate item. Some simple bias indices have been proposed, such as the correlation between the difficulty levels of ability test items in two cultures, or between the levels of endorsement of questionnaire items. Another index of item bias is the interaction between items and culture in an analysis of variance. Psychometrically more sophisticated techniques are based on item response theory or loglinear models. A popular technique for analyzing bias in dichotomous items (e.g., yes–no; right–wrong) is the Mantel–Haenszel procedure (Holland and Wainer, 1993). For items with a graded response scale better use is made of the

Sources of cultural bias 9

available information if an analysis of variance is carried out with item score as the dependent variable and score level on the instrument and culture as independ- ent variables (for overviews of procedures see Van de Vijver and Tanzer, 2004, and Sireci, 2011). It is important to note that analysis techniques as mentioned examine relation- ships between items within an instrument. With these techniques for internal bias analysis, it is quite possible that method bias (affecting all or most items in an instrument) will remain undetected. For the analysis of method bias, external stan- dards or criteria are needed. In culture-comparative studies where relevant com- mon criterion variables are hard to find, the analysis of method bias requires considerable effort, because usually extensive additional data have to be obtained. In Additional Topics Table 12.2 sources of method bias are the most numerous. This reflects the importance of this category of bias, which is often ignored in cross-cultural research. The main measure against effects of method bias is the standardization of data so that the score distributions in each sample have a mean of 0.0 and a standard deviation of 1.0 (e.g., Leung and Bond, 1989) prior to further

Additional Topics Table 12.2 An overview of three types of bias and their possible sources Kind of bias Source Construct Incomplete overlap of definitions of the construct across cultures Differential appropriateness of item content (e.g., skills do not belong to the repertoire of either cultural group) Poor sampling of all relevant behaviors (e.g., short instruments covering broad constructs) Incomplete coverage of the psychological construct Method Differential social desirability Differential response styles such as extremity scoring and acquiescence Differential stimulus familiarity Lack of comparability of samples (e.g., differences in educational background, age or gender composition) Differences in physical testing conditions Differential familiarity with response procedures Tester effects Communication problems between subject and tester in either cultural group Item Poor item translation Inadequate item formulation (e.g., complex wording) One or a few items may invoke additional traits or abilities Incidental differences in appropriateness of the item content (e.g., topic of item of educational test not in curriculum in one cultural group) From Van de Vijver and Poortinga (1997).

Understanding Cultural Bias in Psychometric Assessments: Item, Method, and Construct, Lecture notes of Communication

Related documents

Partial preview of the text

Download Understanding Cultural Bias in Psychometric Assessments: Item, Method, and Construct and more Lecture notes Communication in PDF only on Docsity!

12C Sources of cultural bias