irtQ (version 0.2.1)

cac_rud: Classification accuracy and consistency using Rudner's (2001, 2005) approach.

Description

This function computes the classification accuracy and consistency based on the methods proposed by Rudner (2001, 2005). It can handle both situations where the empirical ability distribution of the population is available and when individual ability estimates are available.

Usage

cac_rud(cutscore, theta = NULL, se, weights = NULL)

Value

A list containing the following elements:

• confusion: A confusion matrix showing the cross table between true and expected levels.

• marginal: A data frame showing the marginal classification accuracy and consistency indices.

• conditional: A data frame showing the conditional classification accuracy and consistency indices.

• prob.level: A data frame showing the probability of being assigned to each level category.

• cutscore: A numeric vector showing the cut scores used in the analysis.

Arguments

cutscore

A numeric vector specifying the cut scores for classification. Cut scores are the points that separate different performance categories (e.g., pass vs. fail, or different grades).

theta

A numeric vector of ability estimates. Ability estimates (theta values) are the individual proficiency estimates obtained from the IRT model. The theta parameter is optional and can be NULL.

se

A numeric vector of the same length as theta indicating the standard errors of the ability estimates.

weights

An optional two-column data frame or matrix where the first column is the quadrature points (nodes) and the second column is the corresponding weights. This is typically used in quadrature-based IRT analysis.

Author

Hwanggyu Lim

Details

The function first checks the provided inputs for correctness. It then computes the probabilities that an examinee with a specific ability is assigned to each level category, and calculates the conditional classification accuracy and consistency for each theta value. Finally, it computes the marginal accuracy and consistency.

References

Rudner, L. M. (2001). Computing the expected proportions of misclassified examinees. Practical Assessment, Research, and Evaluation, 7(1), 14.

Rudner, L. M. (2005). Expected classification accuracy. Practical Assessment, Research, and Evaluation, 10(1), 13.

See Also

gen.weight, est_score, cac_lee

Examples

# \donttest{
## ------------------------------------------------------------------------------
# 1. When the empirical ability distribution of the population is available
## ------------------------------------------------------------------------------
## import the "-prm.txt" output file from flexMIRT
flex_prm <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ")

# read item parameter data and transform it to item metadata
x <- bring.flexmirt(file = flex_prm, "par")$Group1$full_df

# set the cut scores on the theta score metric
cutscore <- c(-2, -0.5, 0.8)

# create the data frame including the quadrature points
# and the corresponding weights
node <- seq(-4, 4, 0.25)
weights <- gen.weight(dist = "norm", mu = 0, sigma = 1, theta = node)

# compute the conditional standard errors across all quadrature points
tif <- info(x = x, theta = node, D = 1, tif = TRUE)$tif
se <- 1 / sqrt(tif)

# calculate the classification accuracy and consistency
cac_1 <- cac_rud(cutscore = cutscore, se = se, weights = weights)
print(cac_1)

## ------------------------------------------------------------------------------
# 2. When individual ability estimates are available
## ------------------------------------------------------------------------------
# randomly select the true abilities from N(0, 1)
set.seed(12)
theta <- rnorm(n = 1000, mean = 0, sd = 1)

# simulate the item response data
data <- simdat(x = x, theta = theta, D = 1)

# estimate the ability parameters and standard errors using the ML estimation
est_theta <- est_score(
  x = x, data = data, D = 1, method = "ML",
  range = c(-4, 4), se = TRUE
)
theta_hat <- est_theta$est.theta
se <- est_theta$se.theta

# calculate the classification accuracy and consistency
cac_2 <- cac_rud(cutscore = cutscore, theta = theta_hat, se = se)
print(cac_2)
# }