content_validity_index: Content Validity Index From Expert Ratings

Description

Quantifies how well a pool of candidate items covers the construct it is meant to measure, using the relevance ratings of a panel of subject matter experts. Validation begins before any respondent data are collected: experts rate each item for relevance, and the content validity index summarizes their agreement (Lynn, 1986; Polit & Beck, 2006; Bandalos, 2018). The function returns the item level index (I-CVI) with an exact binomial confidence interval, the chance corrected modified kappa of Polit, Beck, and Owen (2007), Lawshe's (1975) content validity ratio, and the two scale level summaries S-CVI/Ave and S-CVI/UA. The confidence interval is what keeps a small panel from being read as more informative than it is: with five experts an I-CVI of 0.80 carries an interval roughly seven tenths of the width of the scale.

Usage

content_validity_index(
  ratings,
  relevant = c(3, 4),
  essential = NULL,
  conf_level = 0.95
)

Value

A data.frame (class dmar_tbl) with one row per item, in the order the items appear in ratings, and columns:

item: The item name, from the row names of ratings when present.
n_experts: Number of experts who rated the item (missing ratings excluded).
n_relevant: Number of those experts whose rating was in relevant.
i_cvi: The item level content validity index, n_relevant / n_experts.
ci_lower, ci_upper: Limits of the exact binomial confidence interval for i_cvi at conf_level.
kappa: The modified kappa of Polit, Beck, and Owen (2007), the I-CVI adjusted for chance agreement.
cvr: Lawshe's content validity ratio, computed from essential when supplied and from relevant otherwise.

Attributes: "s_cvi_ave", the mean of the i_cvi column;

"s_cvi_ua", the proportion of items with i_cvi equal to 1;

"relevant", the rating values counted as relevant;

"essential", the rating values counted as essential for the content validity ratio (equal to "relevant" when essential was

NULL); and "conf_level", the confidence level used.

Arguments

ratings: A matrix or data.frame of expert ratings with items in rows and experts in columns, on the conventional 4 point relevance scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant). Every non-missing entry must be one of 1, 2, 3, or 4. Row names, when present, name the items; otherwise items are labeled item_1, item_2, and so on. At least 1 item and at least 2 experts are required. NA is allowed and means that expert did not rate that item; a missing entry lowers that item's expert count and thereby the denominator of its I-CVI, kappa, and CVR, leaving other items untouched. A row with no ratings at all is an error.
relevant: The rating values counted as relevant. Default c(3, 4), the standard dichotomization of the 4 point scale into relevant (3 or 4) versus not relevant (1 or 2). Must be a subset of 1:4.
essential: Optional. The rating values counted as “essential” for Lawshe's content validity ratio, when the panel answered the essential-versus-not question on a separate part of the scale. Must be a subset of 1:4. When NULL (default), the content validity ratio is computed from the same dichotomization as relevance, that is from relevant.
conf_level: Confidence level for the exact binomial interval on each I-CVI. Default 0.95. Must be in (0, 1).

Author

Ken Kelley [email protected]

Details

Let $N$ be the number of experts who rated an item and $A$ the number of those who rated it relevant.

Item level index. The item level content validity index is the proportion of rating experts who called the item relevant, $$\mathrm{I\mbox{-}CVI} = A / N.$$ Lynn (1986) gives the conventional criteria: with five or fewer experts an item is expected to reach 1.00, and with six to ten experts at least 0.78.

Confidence interval. The I-CVI is a binomial proportion, so the interval reported here is the exact (Clopper-Pearson) interval from binom.test at conf_level. Expert panels are small by design and the interval states plainly how little a handful of ratings pins down the population proportion. This is the accuracy in parameter estimation view of content validity: if the interval is too wide to act on, the remedy is more experts.

Chance corrected agreement. Some of the observed agreement on relevance would occur if experts responded at random with probability 0.5, so Polit, Beck, and Owen (2007) correct the index in the manner of a kappa. The probability of chance agreement is the binomial point probability $$p_c = \frac{N!}{A!\,(N - A)!}\, 0.5^{N},$$ and the modified kappa is $$\kappa = \frac{\mathrm{I\mbox{-}CVI} - p_c}{1 - p_c}.$$ The binomial coefficient is evaluated on the log scale with lchoose and then exponentiated, so a large panel does not overflow the way factorial() would.

Content validity ratio. Lawshe (1975) asked a panel whether each item measures behavior that is essential to the performance domain. With $n_e$ experts calling the item essential, $$\mathrm{CVR} = \frac{n_e - N / 2}{N / 2},$$ which equals 1 when every expert says essential, 0 when exactly half do, and -1 when none do.

Scale level summaries. S-CVI/Ave is the mean of the I-CVIs over items, the averaging approach Polit and Beck (2006) recommend reporting. S-CVI/UA is the universal agreement proportion, the fraction of items whose I-CVI equals 1, a stricter and considerably more conservative summary.

References

Bandalos, D. L. (2018). Measurement theory and applications for the social sciences. Guilford Press.

Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563--575.

Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382--385.

Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing and Health, 29(5), 489--497. tools:::Rd_expr_doi("10.1002/nur.20147")

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30(4), 459--467. tools:::Rd_expr_doi("10.1002/nur.20199")

Examples

Run this code

# Six experts rate five candidate items on the 4 point relevance scale.
ratings <- rbind(
  item_1 = c(4, 4, 3, 4, 4, 3),
  item_2 = c(4, 3, 4, 4, 3, 2),
  item_3 = c(2, 3, 1, 2, 3, 2),
  item_4 = c(4, 4, 4, 4, 4, 4),
  item_5 = c(3, 4, 4, 3, NA, 4))
colnames(ratings) <- paste0("expert_", 1:6)
cvi <- content_validity_index(ratings)
cvi

# The scale level summaries travel with the table as attributes.
attr(cvi, "s_cvi_ave")
attr(cvi, "s_cvi_ua")

# The worked example of Polit, Beck, and Owen (2007): 6 experts, 5 of whom
# rate the item relevant, gives I-CVI = 0.83, p_c = 0.094, and a modified
# kappa of 0.816 (the paper reports 0.81, carrying its rounded I-CVI).
content_validity_index(matrix(c(4, 4, 3, 4, 3, 1), nrow = 1))

# A wide interval is the point: with 5 experts an I-CVI of 0.80 is
# compatible with a population proportion anywhere from about 0.28 to 0.99.
content_validity_index(matrix(c(4, 4, 3, 4, 1), nrow = 1))

# The broom verbs on the earlier result: one row per item, and the
# scale-level summary.
generics::tidy(cvi)
generics::glance(cvi)

Run the code above in your browser using DataLab