Learn R Programming

catR (version 3.4)

randomCAT: Random generation of adaptive tests (dichotomous and polytomous models)

Description

This command generates a response pattern to an adaptive test, for a given item bank (with either dichotomous or polytomous models), a true ability level, and several lists of CAT parameters (starting items, stopping rule, provisional and final ability estimators).

Usage

randomCAT(trueTheta, itemBank, model = NULL, responses = NULL, genSeed = NULL, 
 	maxItems = 50, cbControl = NULL, nAvailable = NULL, start = 
 	list(fixItems = NULL, seed = NULL, nrItems = 1, theta = 0, D = 1, halfRange = 2, 
 	startSelect = "MFI"), test = list(method = "BM", priorDist = "norm", 
 	priorPar = c(0, 1), range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), 
 	itemSelect = "MFI", infoType = "observed", randomesque = 1, AP = 1), 
 	stop = list(rule = "length", thr = 20, alpha = 0.05), final = 
 	list(method = "BM", priorDist = "norm", priorPar = c(0, 1), range = c(-4, 4), 
 	D = 1,parInt = c(-4, 4, 33), alpha = 0.05), allTheta = FALSE, 
 	save.output = FALSE, output = c("path", "name", "csv"))
## S3 method for class 'cat':
print(x, ...)
## S3 method for class 'cat':
plot(x, ci = FALSE, alpha = 0.05, trueTh = TRUE, classThr = NULL, 
 	save.plot = FALSE, save.options = c("path", "name", "pdf"),...)

Arguments

trueTheta
numeric: the value of the true ability level.
itemBank
numeric: a suitable matrix of item parameters (possibly augmented by group membership for content balancing). See Details.
model
either NULL (default) for dichotomous models, or any suitable acronym for polytomous models. Possible values are "GRM", "MGRM", "PCM", "GPCM", "RSM" and "NRM". See
responses
either NULL (default) or a vector of pre-specified item responses with as many components as the rows of itemBank. See Details.
genSeed
either a numeric value to fix the random generation of responses pattern or NULL (default). Ignored if responses is not NULL. See Details.
maxItems
numeric: the maximal number of items to be administered (default is 50).
cbControl
either a list of accurate format to control for content balancing, or NULL. See Details.
nAvailable
either a boolean vector indicating which items (denoted by 1's) are available at the start of the test and which (denoted by 0's) are not, or NULL (default). See Details.
start
a list with the options for starting the adaptive test. See Details.
test
a list with the options for provisional ability estimation and next item selection. See Details.
stop
a list with the options of the stopping rule. See Details.
final
a list with the options for final ability estimation. See Details.
allTheta
logical: should all provisional ability estimates and standard errors be computed and returned (including among the starting items)? Default is FALSE, meaning that provisional ability estimates and standard errors are computed only after the
save.output
logical: should the output be saved in an external text file? (default is FALSE).
output
character: a vector of three components. The first component is either the file path to save the output of "path" (default), the second component is the name of the output file, and the third component is the file type, either "txt"
x
an object of class "cat", typically an output of randomCAT function.
ci
logical: should the confidence intervals be plotted for each provisional ability estimate? (default is TRUE).
alpha
numeric: the significance level for provisional confidence intervals (default is 0.05). Ignored if ci is FALSE.
trueTh
logical: should the true ability level be drawn by a horizontal line? (default is TRUE).
classThr
either a numeric value giving the classification threshold to be displayed, or NULL.
save.plot
logical: should the plot be saved in an external figure? (default is FALSE).
save.options
character: a vector of three components. The first component is either the file path or "path" (default), the second component is the name of the output file or ,"name" (default), and the third component is the file extension, ei
...
other generic arguments to be passed to print and plot functions.

Value

  • The function randomCAT returns a list of class "cat" with the following arguments:
  • trueThetathe value of the trueTheta argument.
  • modelthe value of the model argument.
  • maxItemsthe value of the maxItems argument.
  • testItemsa vector with the items that were administered during the test.
  • itemPara matrix with the parameters of the items administered during the test.
  • patternthe generated response pattern (as vector of 0 and 1 entries).
  • thetaProva vector with the provisional ability estimates.
  • seprova vector with the standard errors of the provisional ability estimates.
  • thFinalthe final ability estimate.
  • seFinalthe standrad error of the final ability estimate.
  • ciFinalthe confidence interval of the final ability estimate.
  • genSeedthe value of the genSeed argument.
  • startFixItemsthe value of the start$fixItems argument (or its default value if missing).
  • startSeedthe value of the start$seed argument (or its default value if missing).
  • startNrItemsthe value of the start$nrItems argument (or its default value if missing).
  • startThetathe value of the start$theta argument (or its default value if missing).
  • startDthe value of the start$D argument (or its default value if missing).
  • startHalfRangethe value of the start$halfRange argument (or its default value if missing).
  • startThStartthe starting ability values used for selecting the first items of the test.
  • startSelectthe value of the start$startSelect argument (or its default value if missing).
  • provMethodthe value of the test$method argument (or its default value if missing).
  • provDistthe value of the test$priorDist argument (or its default value if missing).
  • provParthe value of the test$priorPar argument (or its default value if missing).
  • provRangethe value of the test$range argument (or its default value if missing).
  • provDthe value of the test$D argument (or its default value if missing)or NA if model is not NULL.
  • itemSelectthe value of the test$itemSelect argument (or its default value if missing).
  • infoTypethe value of the test$infoType argument (or its default value if missing). Not returned if model is not NULL.
  • randomesquethe value of the test$randomesque argument (or its default value if missing).
  • APthe value of the test$AP argument (or its default value if missing).
  • cbControlthe value of the cbControl argument (or its default value if missing).
  • cbGroupthe value of the itemBank$cbGroup element of the item bank itemBank (for dichotomous IRT models), or the cbGroup element returned by the breakBank function (for polytomous IRT models), or NULL.
  • stopRulethe value of the stop$rule argument (or its default value if missing).
  • stopThrthe value of the stop$thr argument (or its default value if missing).
  • stopAlphathe value of the stop$alpha argument (or its default value if missing).
  • endWarninga logical indactor indicating whether the adaptive test stopped because the stopping rule was satisfied or not.
  • finalMethodthe value of the final$method argument (or its default value if missing).
  • finalDistthe value of the final$priorDist argument (or its default value if missing).
  • finalParthe value of the final$priorPar argument (or its default value if missing).
  • finalRangethe value of the final$range argument (or its default value if missing).
  • finalDthe value of the final$D argument (or its default value if missing), or NA if model is not NULL.
  • finalAlphathe value of the final$alpha argument (or its default value if missing).
  • save.outputthe value of the save.output argument.
  • outputthe value of the output argument.
  • assigned.responsesa logical value, being TRUE if responses was provided or FALSE responses was set to NULL.
  • The function print.cat returns similar (but differently organized) results.

code

Examples

itemize

  • rule: a character string specifying the type of stopping rule. Possible values are:"length"(default), to stop the test after a pre-specified number of items administered;"precision", to stop the test when the provisional standard error of ability becomes less than or equal to the pre-specified value; and"classification", for which the test ends whenever the provisional confidence interval (set by thealphaargument) does not hold the classification threshold anymore (this is also called the ACI rule; see e.g. Thomson, 2009).

item

  • seed: either a numeric value to fix the random seed for item selection, NA to randomly select the items withour fixing the random seed, or NULL (default) to select the items on the basis of their difficulty level. Ignored if fixItems is not NULL.
  • nrItems: the number of first items to be selected (default is 1). Can be set to zero (see below). Ignored if fixItems is not NULL.
  • theta: the central initial ability value, used to define the range of ability levels for selecting the first items (default is 0). Ignored if either fixItems or seed is not NULL. See startItems for further details.
  • D: numeric, the metric constant. Default is D=1 (for logistic metric); D=1.702 yields approximately the normal metric (Haley, 1952). Ignored if model is not NULL and if startSelect is not "MFI".
  • halfRange: the half range of starting ability levels for selecting the first items (default is 2). Ignored if either fixItems or seed is not NULL. See startItems for further details.
  • startSelect: the method for selecting the first items of the test, with possible values "bOpt" and "MFI" (default). Ignored if either fixItems or seed is not NULL. See startItems for further details.
  • priorDist: a character string which sets the prior distribution. Possible values are: "norm" (default) for normal distribution, "unif" for uniform distribution, and "Jeffreys" for Jeffreys' noninformative prior distribution (Jeffreys, 1939, 1946). ignored if method is neither "BM" nor "EAP".
  • priorPar: a vector of two numeric components, which sets the parameters of the prior distribution. If (method="BM" or method=="EAP") and priorDist="norm", the components of priorPar are respectively the mean and the standard deviation of the prior normal density. If (method="BM" or method="EAP") and priorDist="unif", the components of priorPar are respectively the lower and upper bound of the prior uniform density. Ignored in all other cases. By default, priorPar takes the parameters of the prior standard normal distribution (i.e., priorPar=c(0,1)). In addition, priorPar also provides the prior parameters for the comoutation of MLWI and MPWI values for next item selection (see nextItem for further details).
  • range: the maximal range of ability levels, set as a vector of two numeric components. The ability estimate will always lie to this interval (set by default to [-4, 4]). Ignored if method=="EAP".
  • D: the value of the metric constant. Default is D=1 for logistic metric. Setting D=1.702 yields approximately the normal metric (Haley, 1952). Ignored if model is not NULL.
  • parInt: a numeric vector of three components, holding respectively the values of the arguments lower, upper and nqp of the eapEst, eapSem and MWI commands. It specifies the range of quadrature points for numerical integration, and is used for computing the EAP estimate, its standard error, and the MLWI and MPWI values for next item selection. Default vector is (-4, 4, 33), thus setting the range from -4 to 4 by steps of 0.25. Ignored if method is not "EAP" and if itemSelect is neither "MLWI" nor "MPWI".
  • itemSelect: the rule for next item selecion, with possible values "MFI" (default) for maximum Fisher information criterion; "bOpt" for optimal ability-difficulty match (or Urry's procedure) (not available if model is not NULL); "thOpt" for optimal theta selection (not available if model is not NULL); "MLWI" and "MPWI" for respectively maximum likelihood and posterior weighted information criterion; "MEPV" for minimum expected posterior variance; "MEI" for maximum expected information; "KL" and "KLP" for Kullback-Leibler and posterior Kullback-Leibler information methods; "progressive" and "proportional" for progressive and proportional methods; ; and "random" for random selection. For further details, see nextItem.
  • AP: the acceleration parameter required for progressive and proportional methods, with default value 1. Ignored with all other selection methods.
  • infoType: character: the type of information function to be used for next item selection. Possible values are "observed" (default) for observed information function, and "Fisher" for Fisher information function. Ignored if itemselect is not "MEI".
  • randomesque: integer: the number of items to be chosen from the next item selection rule, among those the next item to be administered will be randomly picked up. Default value is 1 and leads to usual selection of the optimal item (Kingsbury and Zara, 1989).
  • thr: a numeric value fixing the threshold of the stopping rule. If rule="length", thr is the maximal number of items to be administered (in practice, it is replaced by the value of the maxItems argument if the latter is smaller than thr). If rule="precision", thr is the precision level (i.e. the standard error) to be reached before stopping. Finally, if rule="classification", thr corresponds to the ability level which serves as a classification rule (i.e. which must not be covered by the provisional confidence interval). The "classification" rule is not available for the progressive and proportional item selection rules.
  • alpha: the significance (or $\alpha$) level for computing the priovisional confidence interval of ability. Ignored if rule is not "classification".

eqn

$se(\hat{\theta})$

deqn

$$[\hat{\theta}-z_{1-\alpha/2} \; se(\hat{\theta}) ; \hat{\theta}+z_{1-\alpha/2} \; se(\hat{\theta})]$$

emph

  • k
  • (k-1)
  • (k-1)
  • not

bold

Examples

Details

The randomCAT function generates an adaptive test using an item bank specified by arguments itemBank and model, and for a given true ability level specified by argument trueTheta. The maximal length of the test can be fixed through the maxItems argument, with a default value of 50 items. Dichotomous IRT models are considered whenever model is set to NULL (default value). In this case, itemBank must be a matrix with one row per item and four columns, with the values of the discrimination, the difficulty, the pseudo-guessing and the inattention parameters (in this order). These are the parameters of the four-parameter logistic (4PL) model (Barton and Lord, 1981). Polytomous IRT models are specified by their respective acronym: "GRM" for Graded Response Model, "MGRM" for Modified Graded Response Model, "PCM" for Partical Credit Model, "GPCM" for Generalized Partial Credit Model, "RSM" for Rating Scale Model and "NRM" for Nominal Response Model. The itemBank still holds one row per item, end the number of columns and their content depends on the model. See genPolyMatrix for further information and illustrative examples of suitable polytomous item banks. By default all item responses will be randomly drawn from parent distribution set by the item bank parameters of the itemBank matrix (using the genPattern function for instance). Moreover, the random generation of the item responses can be fixed (for e.g., replication purposes) by assigning some numeric value to the genSeed argument. By default this argument is equal to NULL so the random seed is not fixed (and two successive runs of randomCAT will usually lead to different response patterns). It is possible, however, to provide a full response pattern of previously recorded responses to each of the item bank, for instance for post-hoc simulations. This is done by providing to the responses argument a vector of binary entries (without missing values). By default responses is set to NULL and item responses will be drawn from the item bank parameters. With the aforementioned item bank structures, content balancing cannot be controled and cbControl must be set to NULL (default value). Otherwise this will most often lead to an error. In order to allow for content balancing control:
  1. theitemBankmust be updated with an additional column holding the group names;
the cbControl argument must be set properly as a list with group names and theoretical proportions for content balancing.

References

Barrada, J. R., Olea, J., Ponsoda, V., and Abad, F. J. (2010). A method for the comparison of item selection rules in computerized adaptive testing. Applied Psychological Measurement, 20, 213-229. doi: 10.1177/0146621610370152 Barton, M.A., and Lord, F.M. (1981). An upper asymptote for the three-parameter logistic item-response model. Research Bulletin 81-20. Princeton, NJ: Educational Testing Service. Birnbaum, A. (1969). Statistical theory for logistic mental test models with a prior distribution of ability. Journal of Mathematical Psychology, 6, 258-276. doi: 10.1016/0022-2496(69)90005-4 Bock, R. D., and Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444. doi: 10.1177/014662168200600405 Haley, D.C. (1952). Estimation of the dosage mortality relationship when the dose is subject to error. Technical report no 15. Palo Alto, CA: Applied Mathematics and Statistics Laboratory, Stanford University. Jeffreys, H. (1939). Theory of probability. Oxford, UK: Oxford University Press. Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, 186, 453-461. Kingsbury, G. G., and Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359-375. doi: 10.1207/s15324818ame0204_6 Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum. Magis, D., and Raiche, G. (2012). Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR. Journal of Statistical Software, 48 (8), 1-31. URL http://www.jstatsoft.org/v48/i08/ Thompson, N. A. (2009). Item selection in computerized classification testing. Educational and Psychological Measurement, 69, 778-793. doi: 10.1177/0013164408324460 Urry, V. W. (1970). A Monte Carlo investigation of logistic test models. Unpublished doctoral dissertation. West Lafayette, IN: Purdue University. van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201-216. doi: 10.1007/BF02294775 Veerkamp, W. J. J., and Berger, M. P. F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22, 203-226. doi: 10.3102/10769986022002203 Warm, T.A. (1989). Weighted likelihood estimation of ability in item response models. Psychometrika, 54, 427-450. doi: 10.1007/BF02294627

See Also

testList, startItems, nextItem, thetaEst, semTheta, eapEst, eapSem, MWI, MEI, KL, breakBank, genPolyMatrix, genPattern

Examples

Run this code
## Dichotomous models ##

 # Loading the 'tcals' parameters 
 data(tcals)

 # Matrix of item parameters (with no content balancing)
 bank <- as.matrix(tcals[,1:4])
 
 # Creation of a starting list: 5 items, initial theta 0, bw 2
 start <- list(nrItems = 5, theta = 0, halfRange = 2)

 # Creation of 'test' list: weighted likelihood estimation of provisional ability, 
 # and MFI criterion for next item selection
 test <- list(method = "WL", itemSelect = "MFI")

 # Creation of 'final' list: EAP estimation of final ability
 final <- list(method = "EAP")

 # Creation of a stopping rule: precision criterion, standard error to be reached 0.3
 stop <- list(rule = "precision", thr = 0.3)

 # CAT test
 res <- randomCAT(0, bank, start = start, test = test, stop = stop, final = final)

 # Creation of a (purely artificial) response pattern for post-hoc simulation
 resp <- rbinom(nrow(bank), 1, 0.5)
 res.ph <- randomCAT(0, bank, start = start, test = test, stop = stop, final = final,
        responses = resp)

 # New 'test' and 'final' rules (BM and EAP estimation with Jeffreys' prior, 
 # randomesque value 5)
 test2 <- list(method = "BM", priorDist = "Jeffreys", randomesque = 5)
 final2 <- list(method = "EAP", priorDist = "Jeffreys")

 # New stopping rule: classification criterion, with classification threshold 0 and 
 # alpha level 0.05
 stop2 <- list(rule = "classification", thr = 0, alpha = 0.05)

 # CAT test with new 'test', 'stop' and 'final' rules
 res2 <- randomCAT(0, bank, start = start, test = test2, stop = stop2, final = final2)

 # New stopping rule: classification criterion, with classification threshold 0.5
 # and alpha level 0.05
 stop3 <- list(rule = "classification", thr = 0.5, alpha = 0.05)

 # CAT test with new 'stop' rule
 res3 <- randomCAT(0, bank, start = start, test = test2, stop = stop3, final=final2)

 # new 'test' and 'stop' rule for next item selection
 test3 <- list(method = "WL", itemSelect = "MLWI")
 stop4 <- list(rule = "length",thr = 10)
 res4 <- randomCAT(0, bank, start = start, test = test3, stop = stop4, final = final2)

 # Creation of the 'cbList' list with arbitrary proportions
 cbList <- list(names = c("Audio1", "Audio2", "Written1", "Written2",
        "Written3"), props = c(0.1, 0.2, 0.2, 0.2, 0.3))

 # CAT test with 'start', 'test', 'stop4' and 'final2' lists and content balancing 
 # using 'cbList' ('tcals' must be used now for content balancing)
 res5 <- randomCAT(0, tcals, start = start, test = test, stop = stop4, final = final2, 
      cbControl = cbList)

 # Using progressive item selection rule and requiring all ability estimates and SEs
 test4 <- list(itemSelect = "progressive")
 res6 <- randomCAT(0, tcals, start = start, test = test4, stop = stop4, final = final, 
      cbControl = cbList, allTheta = TRUE)

 # Saving the output in the external 'out' text file within folder 'Program Files' 
 # of hard drive 'C'
 res5 <- randomCAT(0, tcals, start = start, test = test, stop = stop4, final = final2, 
      cbControl = cbList, save.output = TRUE, 
      output = c("c:/Program Files/", "out", "txt"))

 # Plotting results
 plot(res)
 plot(res, ci = TRUE)
 plot(res, ci = TRUE, trueTh = FALSE)
 plot(res, ci = TRUE, classThr = 1)

 # Saving last figure into PDF file 'figure' within folder 'C:/Program Files/'
 plot(res, ci = TRUE, classThr = 1, save.plot = TRUE,
      save.options = c("c:/Program Files/", "figure", "pdf"))

 # With mistake
 plot(res, ci = 0.05)
 plot(res, classThr = TRUE)


## Polytomous models ##

 # Generation of an item bank under GRM with 100 items and at most 4 categories
 m.GRM <- genPolyMatrix(100, 4, "GRM")
 m.GRM <- as.matrix(m.GRM)

 # CAT options
 start <- list(nrItems = 3,startSelect = "MFI")
 test <- list(method = "BM", itemSelect = "KL")
 stop <- list(rule = "precision", thr = 0.4)
 final <- list(method = "EAP")

 # CAT test
 res <- randomCAT(0, m.GRM,model = "GRM", start = start, test = test, stop = stop,
 	final = final)
 res

 # Creation of an appropriate list for content balancing
 # Equal proprotions across subgroups of items
 cbList <- list(names = c("Group1", "Group2", "Group3", "Group4"), props = rep(1,4))

 # With content balancing, all ability estimates and progressive method
 m.GRM <- genPolyMatrix(100, 4, "GRM", cbControl = cbList)
 test <- list(method = "BM", itemSelect = "progressive")
 res <- randomCAT(0, m.GRM, model = "GRM", start = start, test = test, stop = stop, 
 	final = final, cbControl = cbList, allTheta = TRUE)
 res

 # Generation of an item bank under PCM with 100 items, 4 categories and groups
 m.PCM <- genPolyMatrix(100, 4, "PCM", same.nrCat = TRUE, cbControl = cbList)
 stop <- list(rule = "length", thr = 10)
 res <- randomCAT(0, m.PCM, model = "PCM", start = start, test = test, stop = stop, 
 	final = final, cbControl = cbList, allTheta = TRUE)
 res

Run the code above in your browser using DataLab