aberrance (version 0.2.1)

detect_tt: Detect test tampering

Description

Detect test tampering at the person level or at the group level.

Usage

detect_tt(
  method,
  psi,
  xi = NULL,
  xi_c = NULL,
  xi_s = NULL,
  x = NULL,
  d = NULL,
  r = NULL,
  x_0 = NULL,
  d_0 = NULL,
  r_0 = NULL,
  interval = c(-4, 4),
  alpha = 0.05,
  group = NULL,
  c = -0.5
)

Value

A list is returned with the following elements:

stat

A matrix of test tampering detection statistics.

pval

A matrix of p-values.

flag

An array of flagging results. The first dimension corresponds to persons/groups, the second dimension to methods, and the third dimension to significance levels.

Arguments

method

The test tampering statistic(s) to compute. Options for score and distractor-based statistics are:

• "EDI_SD_*" for the erasure detection index (Wollack et al., 2015; Wollack & Eckerly, 2017).

• "GBT_SD" for the generalized binomial test statistic (Sinharay & Johnson, 2017). Note: This statistic cannot be computed at the group level.

• "L_SD" for the signed likelihood ratio test statistic (Sinharay et al., 2017). Note: This statistic cannot be computed at the group level.

Options for response-based statistics are:

• "EDI_R_*" for the erasure detection index (Wollack et al., 2015; Wollack & Eckerly, 2017).

• "GBT_R" for the generalized binomial test statistic (Sinharay & Johnson, 2017). Note: This statistic cannot be computed at the group level.

• "L_R" for the signed likelihood ratio test statistic (Sinharay et al., 2017). Note: This statistic cannot be computed at the group level.

Statistics ending in "*" can be computed using various corrections. Options are:

• "*" for all possible corrections.

• "NO" for no correction (Sinharay, 2018; Wollack et al., 2015).

• "CO" for the continuity correction (Wollack et al., 2015; Wollack & Eckerly, 2017). The value of the continuity correction can be specified using c.

• "TS" for the Taylor series expansion (Sinharay, 2018).

psi

A matrix of item parameters.

xi, xi_c, xi_s

Matrices of person parameters. xi is based on all items, xi_c is based on items with changed answers, and xi_s is based on items with the same answers. If NULL (default), person parameters are estimated using maximum likelihood estimation.

x, d, r

Matrices of final data. x is for the item scores, d the item distractors, and r the item responses.

x_0, d_0, r_0

Matrices of initial data. x_0 is for the item scores, d_0 the item distractors, and r_0 the item responses.

interval

The interval to search for the person parameters. Default is c(-4, 4).

alpha

Value(s) between 0 and 1 indicating the significance level(s) used for flagging. Default is 0.05.

group

A vector indicating group membership. If NULL (default), statistics are computed at the person level.

c

Use with the erasure detection index. A value indicating the continuity correction. Default is -0.5.

References

Sinharay, S., Duong, M. Q., & Wood, S. W. (2017). A new statistic for detection of aberrant answer changes. Journal of Educational Measurement, 54(2), 200--217.

Sinharay, S., & Johnson, M. S. (2017). Three new methods for analysis of answer changes. Educational and Psychological Measurement, 77(1), 54--81.

Sinharay, S. (2018). Detecting fraudulent erasures at an aggregate level. Journal of Educational and Behavioral Statistics, 43(3), 286--315.

Wollack, J. A., Cohen, A. S., & Eckerly, C. A. (2015). Detecting test tampering using item response theory. Educational and Psychological Measurement, 75(6), 931--953.

Wollack, J. A., & Eckerly, C. A. (2017). Detecting test tampering at the group level. In G. J. Cizek & J. A. Wollack (Eds.), Handbook of quantitative methods for detecting cheating on tests (pp. 214--231). Routledge.

Examples

# Setup for Examples 1 and 2 ------------------------------------------------

# Settings
set.seed(0)     # seed for reproducibility
N <- 500        # number of persons
n <- 40         # number of items
G <- 20         # number of groups

# Create groups
group <- rep(1:G, each = N / G)

# Randomly select 20% tampered groups with 20% tampered persons
cg <- sample(1:G, size = G * 0.20)
cv <- NULL
for (g in cg) {
  cv <- c(cv, sample(which(group == g), size = N / G * 0.20))
}

# Create vectors of indicators (1 = tampered, 0 = non-tampered)
group_ind <- ifelse(1:G %in% cg, 1, 0)
person_ind <- ifelse(1:N %in% cv, 1, 0)

# Generate person parameters for the nested logit model
xi <- MASS::mvrnorm(
  N,
  mu = c(theta = 0.00, eta = 0.00),
  Sigma = matrix(c(1.00, 0.80, 0.80, 1.00), ncol = 2)
)

# Generate item parameters for the nested logit model
psi <- cbind(
  a = rlnorm(n, meanlog = 0.00, sdlog = 0.25),
  b = rnorm(n, mean = 0.00, sd = 1.00),
  c = runif(n, min = 0.05, max = 0.30),
  lambda1 = rnorm(n, mean = 0.00, sd = 1.00),
  lambda2 = rnorm(n, mean = 0.00, sd = 1.00),
  lambda3 = rnorm(n, mean = 0.00, sd = 1.00),
  zeta1 = rnorm(n, mean = 0.00, sd = 1.00),
  zeta2 = rnorm(n, mean = 0.00, sd = 1.00),
  zeta3 = rnorm(n, mean = 0.00, sd = 1.00)
)

# Simulate uncontaminated data
dat <- sim(psi, xi)
x_0 <- x <- dat$x
d_0 <- d <- dat$d

# Simulate 5% random erasures for non-tampered persons
r_0 <- r <- ifelse(x == 1, 4, d)
for (v in setdiff(1:N, cv)) {
  ci <- sample(1:n, size = n * 0.05)
  for (i in ci) {
    r_0[v, i] <- sample((1:4)[-r[v, i]], size = 1)
  }
  x_0[v, ci] <- ifelse(r_0[v, ci] == 4, 1, 0)
  d_0[v, ci] <- ifelse(r_0[v, ci] == 4, NA, r_0[v, ci])
}
rm(r_0, r)

# Modify contaminated data by tampering with 20% of the scores and
# distractors
for (v in cv) {
  ci <- sample(1:n, size = n * 0.20)
  x[v, ci] <- 1
  d[v, ci] <- NA
}

# Example 1: Person-Level Statistics ----------------------------------------

# Detect test tampering
out <- detect_tt(
  method = c("EDI_SD_*", "GBT_SD", "L_SD"),
  psi = psi,
  x = x,
  d = d,
  x_0 = x_0,
  d_0 = d_0
)

# Example 2: Group-Level Statistics -----------------------------------------

# Detect test tampering
out <- detect_tt(
  method = "EDI_SD_*",
  psi = psi,
  x = x,
  d = d,
  x_0 = x_0,
  d_0 = d_0,
  group = group
)