irtQ (version 0.2.1)

cac_lee: Classification accuracy and consistency using Lee's (2010) approach.

Description

This function computes the classification accuracy and consistency indices for complex assessments using the method proposed by Lee (2010). The function can handle both dichotomous and polytomous item response theory (IRT) models.

Usage

cac_lee(x, cutscore, theta = NULL, weights = NULL, D = 1, cut.obs = TRUE)

Value

A list containing the following elements:

• confusion: A confusion matrix showing the cross table between true and expected levels.

• marginal: A data frame showing the marginal classification accuracy and consistency indices.

• conditional: A data frame showing the conditional classification accuracy and consistency indices.

• prob.level: A data frame showing the probability of being assigned to each level category.

• cutscore: A numeric vector showing the cut scores used in the analysis.

Arguments

x

A data frame containing the item metadata (e.g., item parameters, number of score categories, models ...). See est_irt, irtfit, info or simdat for more detail about the item metadata.

cutscore

A numeric vector specifying the cut scores for classification. Cut scores are the points that separate different performance categories (e.g., pass vs. fail, or different grades).

theta

A numeric vector of ability estimates. Ability estimates (theta values) are the individual proficiency estimates obtained from the IRT model. The theta parameter is optional and can be NULL.

weights

An optional two-column data frame or matrix where the first column is the quadrature points (nodes) and the second column is the corresponding weights. This is typically used in quadrature-based IRT analysis.

D

A scaling factor in IRT models to make the logistic function as close as possible to the normal ogive function (if set to 1.7). Default is 1.

cut.obs

A logical value. If TRUE, it indicates the cutscores on the observed-summed score metric. If FALSE, it indicates they are on the IRT theta score metric. Default is TRUE.

Author

Hwanggyu Lim

Details

The function works by first checking the provided inputs. If both theta and weights are NULL, the function will stop and return an error message. Depending on the provided inputs, the function will then compute the classification accuracy and consistency indices using either the quadrature points and corresponding weights (D method) or individual ability estimates (P method). The function returns a list containing the confusion matrix, marginal and conditional classification accuracy and consistency indices, probabilities of being assigned to each level category, and cut scores.

References

Lee, W. C. (2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47(1), 1-17.

See Also

gen.weight, est_score, cac_rud

Examples

# \donttest{
## ------------------------------------------------------------------------------
# 1. When the empirical ability distribution of the population is available
#    (D method)
## ------------------------------------------------------------------------------
## import the "-prm.txt" output file from flexMIRT
flex_prm <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ")

# read item parameter data and transform it to item metadata
x <- bring.flexmirt(file = flex_prm, "par")$Group1$full_df

# set the cut scores on the observed-summed score metric
cutscore <- c(10, 20, 30, 50)

# create the data frame including the quadrature points
# and the corresponding weights
node <- seq(-4, 4, 0.25)
weights <- gen.weight(dist = "norm", mu = 0, sigma = 1, theta = node)

# calculate the classification accuracy and consistency
cac_1 <- cac_lee(x = x, cutscore = cutscore, weights = weights, D = 1)
print(cac_1)

## ------------------------------------------------------------------------------
# 2. When individual ability estimates are available (P method)
## ------------------------------------------------------------------------------
# randomly select the true abilities from N(0, 1)
set.seed(12)
theta <- rnorm(n = 1000, mean = 0, sd = 1)

# simulate the item response data
data <- simdat(x = x, theta = theta, D = 1)

# estimate the ability parameters using the ML estimation
est_th <- est_score(
  x = x, data = data, D = 1, method = "ML",
  range = c(-4, 4), se = FALSE
)$est.theta

# calculate the classification accuracy and consistency
cac_2 <- cac_lee(x = x, cutscore = cutscore, theta = est_th, D = 1)
print(cac_2)

## ------------------------------------------------------------------------------
# 3. When individual ability estimates are available,
#    but, the cutscores are on the IRT theta score metric
## ------------------------------------------------------------------------------
# set the theta cut scures
cutscore <- c(-2, -0.4, 0.2, 1.0)

# calculate the classification accuracy and consistency
cac_3 <- cac_lee(
  x = x, cutscore = cutscore, theta = est_th, D = 1,
  cut.obs = FALSE
)
print(cac_3)
# }