Docsity
Docsity

Prepare for your exams
Prepare for your exams

Study with the several resources on Docsity


Earn points to download
Earn points to download

Earn points by helping other students or get them with a premium plan


Guidelines and tips
Guidelines and tips

Understanding Cultural Bias in Psychometric Assessments: Item, Method, and Construct, Lecture notes of Communication

The concept of cultural bias in psychometric assessments, its sources, and its impact on equivalence. It covers item bias, method bias, and construct bias, and provides examples and techniques for identifying and addressing bias in assessments. The document also emphasizes the importance of understanding cultural differences in the context of cross-cultural research.

Typology: Lecture notes

2021/2022

Uploaded on 09/27/2022

stifler
stifler 🇮🇹

4

(6)

215 documents

1 / 3

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
12C Sources of cultural bias
A lack of equivalence can be due to a host of reasons, which are referred to with
the generic term of cultural bias. The effects of a biasing factor can be limited to
a single item or a few items (e.g., translation errors). This is called item bias or
differential item functioning (DIF). The effects can also infl uence the responses
on most or all items, leading to method bias. Finally, bias can pertain to the op-
erationalization or defi nition of a concept; this is referred to as concept bias or
construct bias. An overview of various sources of bias is presented in Additional
Topics Table 12.2. More extensive overviews can be found in Van de Vijver and
Tanzer (2004) and Van de Vijver and Leung (1997).
It is clear that there is a correspondence between levels of equivalence and
sources of bias. This is most evident at the level of concept bias: unequal notions
about concepts will lead to systematic differences in concept representation across
cultures. The consequences for equivalence are least clear in the case of item bias.
If only a single item or a few items show evidence of bias, they can be eliminated
and this improves the equivalence of scores. However, evidence of item bias can
also be taken as an indication that an instrument does not represent precisely
identical traits.
The notion of item bias is immediately clear if one thinks of an example. An
item asking for the name of the capital of a country should be much easier for na-
tionals of that country than in other countries. However, item bias can also be the
consequence of subtle differences in shades of meaning. An item is identifi ed as
biased if test takers with the same score on the instrument but belonging to differ-
ent populations do not have the same probability of giving a certain answer (e.g.,
“yes” answer, correct answer) for the item (Shepard, Camilli and Averill, 1981).
Thus, the test score variable on the instrument as a whole is used as the standard to
evaluate each separate item. Some simple bias indices have been proposed, such as
the correlation between the diffi culty levels of ability test items in two cultures, or
between the levels of endorsement of questionnaire items. Another index of item
bias is the interaction between items and culture in an analysis of variance.
Psychometrically more sophisticated techniques are based on item response
theory or loglinear models. A popular technique for analyzing bias in dichotomous
items (e.g., yes–no; right–wrong) is the Mantel–Haenszel procedure (Holland and
Wainer, 1993). For items with a graded response scale better use is made of the
9780521745208_c12C.indd 89780521745208_c12C.indd 8 1/29/11 1:01:29 PM1/29/11 1:01:29 PM
pf3

Partial preview of the text

Download Understanding Cultural Bias in Psychometric Assessments: Item, Method, and Construct and more Lecture notes Communication in PDF only on Docsity!

12C Sources of cultural bias

A lack of equivalence can be due to a host of reasons, which are referred to with the generic term of cultural bias. The effects of a biasing factor can be limited to a single item or a few items (e.g., translation errors). This is called item bias or differential item functioning (DIF). The effects can also influence the responses on most or all items, leading to method bias. Finally, bias can pertain to the op- erationalization or definition of a concept; this is referred to as concept bias or construct bias. An overview of various sources of bias is presented in Additional Topics Table 12.2. More extensive overviews can be found in Van de Vijver and Tanzer (2004) and Van de Vijver and Leung (1997). It is clear that there is a correspondence between levels of equivalence and sources of bias. This is most evident at the level of concept bias: unequal notions about concepts will lead to systematic differences in concept representation across cultures. The consequences for equivalence are least clear in the case of item bias. If only a single item or a few items show evidence of bias, they can be eliminated and this improves the equivalence of scores. However, evidence of item bias can also be taken as an indication that an instrument does not represent precisely identical traits. The notion of item bias is immediately clear if one thinks of an example. An item asking for the name of the capital of a country should be much easier for na- tionals of that country than in other countries. However, item bias can also be the consequence of subtle differences in shades of meaning. An item is identified as biased if test takers with the same score on the instrument but belonging to differ- ent populations do not have the same probability of giving a certain answer (e.g., “yes” answer, correct answer) for the item (Shepard, Camilli and Averill, 1981). Thus, the test score variable on the instrument as a whole is used as the standard to evaluate each separate item. Some simple bias indices have been proposed, such as the correlation between the difficulty levels of ability test items in two cultures, or between the levels of endorsement of questionnaire items. Another index of item bias is the interaction between items and culture in an analysis of variance. Psychometrically more sophisticated techniques are based on item response theory or loglinear models. A popular technique for analyzing bias in dichotomous items (e.g., yes–no; right–wrong) is the Mantel–Haenszel procedure (Holland and Wainer, 1993). For items with a graded response scale better use is made of the

Sources of cultural bias 9

available information if an analysis of variance is carried out with item score as the dependent variable and score level on the instrument and culture as independ- ent variables (for overviews of procedures see Van de Vijver and Tanzer, 2004, and Sireci, 2011). It is important to note that analysis techniques as mentioned examine relation- ships between items within an instrument. With these techniques for internal bias analysis, it is quite possible that method bias (affecting all or most items in an instrument) will remain undetected. For the analysis of method bias, external stan- dards or criteria are needed. In culture-comparative studies where relevant com- mon criterion variables are hard to find, the analysis of method bias requires considerable effort, because usually extensive additional data have to be obtained. In Additional Topics Table 12.2 sources of method bias are the most numerous. This reflects the importance of this category of bias, which is often ignored in cross-cultural research. The main measure against effects of method bias is the standardization of data so that the score distributions in each sample have a mean of 0.0 and a standard deviation of 1.0 (e.g., Leung and Bond, 1989) prior to further

Additional Topics Table 12.2 An overview of three types of bias and their possible sources Kind of bias Source Construct Incomplete overlap of definitions of the construct across cultures Differential appropriateness of item content (e.g., skills do not belong to the repertoire of either cultural group) Poor sampling of all relevant behaviors (e.g., short instruments covering broad constructs) Incomplete coverage of the psychological construct Method Differential social desirability Differential response styles such as extremity scoring and acquiescence Differential stimulus familiarity Lack of comparability of samples (e.g., differences in educational background, age or gender composition) Differences in physical testing conditions Differential familiarity with response procedures Tester effects Communication problems between subject and tester in either cultural group Item Poor item translation Inadequate item formulation (e.g., complex wording) One or a few items may invoke additional traits or abilities Incidental differences in appropriateness of the item content (e.g., topic of item of educational test not in curriculum in one cultural group) From Van de Vijver and Poortinga (1997).