autoFC (version 0.2.0.1002)

sa_pairing_generalized: Automatic Item Pairing Method in Forced-Choice Test Construction

Description

Automatic construction of forced-choice tests based on Simulated Annealing algorithm. Allows items to be:

1. Matched in either pairs, triplets, quadruplets or blocks of any size;

2. Matched based on any number of item-level characteristics (e.g. Social desirability, factor) based on any customized criteria;

3. Matched based on person-level inter-item agreement (IIA) metrics.

Usage

sa_pairing_generalized(block, total_items, Temperature,
                              eta_Temperature = 0.01, r = 0.999,
                              end_criteria = 10^(-6),
                              item_chars, weights, FUN,
                              n_exchange = 2, prob_newitem = 0.25,
                              use_IIA = FALSE, rater_chars,
                              iia_weights = c(BPlin = 1, BPquad = 1,
                              AClin = 1, ACquad = 1))

Value

A list containing:

• block_initial Initial starting block

  energy_initial Initial energy for block_initial

  block_final Final paired block after optimization by SA

  energy_final Final energy for block_final

Arguments

block

An n by k integer matrix, where n is the number of item blocks and k is the number of items per block.

Serves as the initial starting blocks for the automatic pairing method.

total_items

Integer value. How many items do we sample from in order to build this block? Should be more than number of unique values in block.

Temperature

Initial temperature value. Can be left blank and be computed based on the absolute value of initial energy of block (Recommended), and scaled by eta_Temperature.

In general, higher temperature represents a higher probability of accepting an inferior solution.

eta_Temperature

A positive numeric value. The ratio of initial temperature to initial energy of block, if Temperature is not designated.

r

A positive numeric value less than 1. Determines the reduction rate of Temperature after each iteration.

end_criteria

A positive numeric value less than 1. Iteration stops when temperature drops to below end_criteria * Temperature. Default to be \(10^-6\).

item_chars

An m by r data frame, where m is the total number of items to sample from, whether it is included in the block or not, whereas r is the number of item characteristics.

weights

A vector of length r with weights for each item characteristics in item_chars. Should provide a weight of 0 for specific characteristics not of interest, such as item ID.

FUN

A vector of customized function names for optimizing each item characteristic within each block, with length r.

n_exchange

Integer value. Determines how many blocks are exchanged in order to produce a new solution for each iteration. Should be a value larger than 1 and less than nrow(block).

prob_newitem

A value between 0 and 1. Probability of choosing the strategy of picking a new item, when not all candidate items are used to build the FC scale.

use_IIA

Logical. Are IIA metrics used when performing automatic pairing?

rater_chars

A p by m numeric matrix with scores of each of the p participants for the m items. Ignored when use_IIA == FALSE.

iia_weights

A vector of length 4 indicating weights given to each IIA metric:

Linearly weighted AC (Gwet, 2008; 2014);

Quadratic weighted AC;

Linearly weighted Brennan-Prediger (BP) Index(Brennan & Prediger, 1981; Gwet, 2014);

Quadratic weighted BP.

Author

Mengtong Li

References

Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687-699. https://doi.org/10.1177/001316448104100307

Gwet, K. L. (2008). Computing inter rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. https://doi.org/10.1348/000711006X126600

Gwet, K. L. (2014). Handbook of inter-rater reliability (4th ed.): The definitive guide to measuring the extent of agreement among raters. Gaithersburg, MD: Advanced Analytics Press.

Examples

## Simulate 60 items loading on different Big Five dimensions,
## with different mean and item difficulty

item_dims <- sample(c("Openness","Conscientiousness","Neuroticism",
                     "Extraversion","Agreeableness"), 60, replace = TRUE)
item_mean <- rnorm(60, 5, 2)
item_difficulty <- runif(60, -1, 1)


item_df <- data.frame(Dimensions = item_dims,
                     Mean = item_mean, Difficulty = item_difficulty)
solution <- make_random_block(60, 60, 3)


item_responses <- matrix(sample(seq(1:5), 600*60, replace = TRUE), nrow = 60, byrow = TRUE)

## Automatic pairing, without use of IIAs
## See ?facfun for information about what it does
# \donttest{
sa_pairing_generalized(solution, 60, eta_Temperature = 0.01,
                                   r = 0.999, end_criteria = 0.001,
                                   weights = c(1,1,1),
                                   item_chars = item_df,
                                   FUN = c("facfun", "var", "var"))
# }

## Automatic pairing, with IIAs
# \donttest{
sa_pairing_generalized(solution, 60, eta_Temperature = 0.01,
                                   r = 0.999, end_criteria = 0.001,
                                   weights = c(1,1,1),
                                   item_chars = item_df,
                                   FUN = c("facfun", "var", "var"),
                                   use_IIA = TRUE,
                                   rater_chars = item_responses,
                                   iia_weights = c(BPlin = 1, BPquad = 1,
                                   AClin = 1, ACquad = 1))
# }