itChoose: Choose the Next Item in a CAT

Description

itChoose chooses the next item in a CAT based on the remaining items and a variety of item selection algorithms.

Usage

itChoose( left_par, mod = c("brm", "grm"), 
          numb = 1, n.select = 1,
          cat_par = NULL, cat_resp = NULL, cat_theta = NULL,
          select = c("UW-FI", "LW-FI", "PW-FI",
                     "FP-KL", "VP-KL", "FI-KL", "VI-KL",
                     "random"),
          at = c("theta", "bounds"),
          range = c(-6, 6), it.range = NULL,
          delta = NULL, bounds = NULL,
          ddist = dnorm, quad = 33, ... )

Arguments

left_par

numeric: a matrix of item parameters from which to choose the next item. The rows must index the items, and the columns must designate the item parameters (in the appropriate order, see

mod

character: a character string indicating the IRT model. Current support is for the 3-parameter binary response model (), and Samejima's graded response model (). The contents

numb

numeric: a scalar indicating the number of items to return to the user. If numb is less than n.select, then itChoose will randomly select numb

n.select

numeric: an integer indicating the number of items to randomly select between at one time. For instance, if select is , at is

cat_par

numeric: either NULL or a matrix of item parameters that have already been administered in the CAT. cat_par only needs to be specified if letting select equal either

cat_resp

numeric: either NULL or a vector of responses corresponding to the items specified in cat_par. cat_par only needs to be specified if letting select equal e

cat_theta

numeric: either NULL or a scalar corresponding to the current ability estimate. cat_theta is not needed if selecting items at or using or

select

character: a character string indicating the desired item selection method. Items can be selected either through maximum Fisher information or Kullback-Leibler divergence methods or randomly. The Fisher information

character: a character string indicating where to select items.

range

numeric: a 2-element numeric vector indicating the range of values that itChoose should average over if select equals or .

it.range

numeric: Either a 2-element numeric vector indicating the minimum and maximum allowed difficulty parameters for selected items (only if mod is equal to ) or NULL indicatin

delta

numeric: a scalar indicating the multiplier used in item selection if a Kullback-Leibler method is chosen. For fixed-point KL divergence, delta is frequently .1 or .2, whereas in variable-point KL div

bounds

numeric: a vector of fixed-points/bounds from which to select items if at equals .

ddist

function: a function indicating how to calculate prior densities if select equals (i.e., weighting Fisher information on the posterior distribution). See

quad

numeric: a scalar indicating the number of quadrature points when select equals or . See Details for more information.

...

arguments passed to ddist, usually distribution parameters identified by name.

Value

itChoose returns a list of the following elements:
paramsa matrix corresponding to the next item or numb items to administer in a CAT with the first column indicating the item number
infoa vector of corresponding information for the numb items of params.
typethe type of information returned in info, which is equal to the item selection algorithm.

Details

The function itChoose returns the next item(s) to administer in a CAT environment. The item selection algorithms fall into three major types: Fisher information, Kullback-Leibler divergence, and random.

If choosing items based on Fisher information (selectequals,, or), then items are selected based on some aggregation of Fisher information (seeFI). The difference between the three Fisher information methods are the weighting functions used (see van der Linden, 1998; Veerkamp & Berger, 1997). Let$$I(w_{ij} | a_j, b_j, c_j) = \int_{-\infty}^{\infty} w_{ij}I_j(\theta)\mu(d\theta)$$be the "average" Fisher information, weighted by real valued function$w_{ij}$. Then all three Fisher information criteria can be explained solely as using different weights. Unweighted Fisher information () sets$w_{ij}$equal to a Dirac delta function with all of its mass either ontheta(ifatequals) or the nearest classification bound (ifatequals). Likelihood-Weighted Fisher information () sets$w_{ij}$equal to the likelihood function given all of the previously administered items (Veerkamp & Berger, 1997). And Posterior-Weighted Fisher information () sets$w_{ij}$equal to the likelihood function times the prior distribution specified inddist(van der Linden, 1998). All three algorithms select items based on maximizing the respective criterion withthe most popular CAT item selection algorithm and equivalent to maximizing Fisher information at a point (Pashley & van der Linden, 2010).
If choosing items based on Kullback-Leibler divergence (
selectequals,,, or), then items are selected based on some aggregation of KL divergence (seeKL).
- The Pointwise KL divergence criteria (selectequalsand) compares KL divergence at two points:$$KL(w_{ij} | a_j, b_j, c_j) = KL_j(P + w_{ij} || P - w_{ij})$$The difference betweenandare the weights used. Fixed Pointwise KL divergence () sets$w_{ij}$equal to, and Variable Pointwise KL divergence () sets$w_{ij}$equal tomultiplied by$1/\sqrt{n}$, where$n$is equal to the number of items given to this point in the CAT (see Chang & Ying, 1996).
- The Integral KL divergence criteria (selectequalsand) integrates KL divergence across a small area:$$KL(w_{ij} | a_j, b_j, c_j) = \int_{P - w_{ij}}^{P + w_{ij}} KL_j(\theta || P) d\theta$$As in Pointwise KL divergence, Fixed Integral KL divergence () sets$w_{ij}$equal to, and Variable Integral KL divergence () sets$w_{ij}$equal tomultiplied by$1/\sqrt{n}$(see Chang & Ying, 1996).
All KL divergence criteria set$P$equal totheta(ifatequals) or the nearest classification bound (ifatequals) and select items based on maximizing the respective criterion.
Ifselectis, thenitChooserandomly picks the next item(s) out of the remaining items in the bank.

References

Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213 -- 229.

Pashley, P. J., & van der Linden, W. J. (2010). Item selection and ability estimation in adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of adaptive testing (pp. 3 -- 30). New York, NY: Springer.

van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201 -- 216.

Veerkamp, W. J. J., & Berger, M. P. F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22, 203 -- 226.

Examples

Run this code

#########################
# Binary Response Model #
#########################
set.seed(888)
# generating an item bank under a binary response model:
b.params <- cbind(a = runif(100, .5, 1.5), b = rnorm(100, 0, 2), c = .2)
# simulating responses using default theta:
b.mod <- simIrt(theta = 0, params = b.params, mod = "brm")

# separating the items into "administered" and "not administered":
left_par <- b.mod$params[1:95, ]
cat_par <- b.mod$params[96:100, ]
cat_resp <- b.mod$resp[ , 96:100]

# running simIrt automatically adds the item numbers to the front!

# attempting each item selection algorithm (except random):
uwfi.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_theta = 0,
                    select = "UW-FI",
                    at = "theta")
lwfi.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_resp = cat_resp,
                    select = "LW-FI")
pwfi.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_resp = cat_resp,
                    select = "PW-FI", ddist = dnorm, mean = 0, sd = 1)

fpkl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_theta = 0,
                    select = "FP-KL",
                    at = "theta", delta = 1.96)
vpkl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_theta = 0,
                    select = "VP-KL",
                    at = "theta", delta = 1.96)
fikl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_theta = 0,
                    select = "FI-KL",
                    at = "theta", delta = 1.96)
vikl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_theta = 0,
                    select = "VI-KL",
                    at = "theta", delta = 1.96)

# which items were the most popular?
uwfi.it$params  # 61 (b close to 0)
lwfi.it$params  # 55 (b close to -2.5)
pwfi.it$params  # 16 (b close to -0.5)
fpkl.it$params  # 61 (b close to 0)
vpkl.it$params  # 61 (b close to 0)
fikl.it$params  # 16 (b close to -0.5)
vikl.it$params  # 16 (b close to -0.5)

# if we pick the top 10 items for "FI-KL":
fikl.it2 <- itChoose(left_par = left_par, mod = "brm",
                     numb = 10, n.select = 10,
                     cat_theta = 0,
                     select = "FI-KL",
                     at = "theta", delta = 1.96)

# we find that item 61 is the third best item
fikl.it2$params

# why did "LW-FI" pick an item with a strange difficulty?
cat_resp

# because cat_resp is mostly 0 ...
# --> so the likelihood is weighted toward negative numbers.

#########################
# Graded Response Model #
#########################
set.seed(999)
# generating an item bank under a graded response model:
g.params <- cbind(runif(100, .5, 1.5), rnorm(100), rnorm(100),
                                       rnorm(100), rnorm(100), rnorm(100))
# simulating responses (so that the parameters are ordered - see simIrt)
left_par <- simIrt(theta = 0, params = g.params, mod = "grm")$params

# now we can choose the best item for theta = 0 according to FI:
uwfi.it2 <- itChoose(left_par = left_par, mod = "brm",
                     numb = 1, n.select = 1,
                     cat_theta = 0,
                     select = "UW-FI",
                     at = "theta")
uwfi.it2

Run the code above in your browser using DataLab