betafunctions (version 1.2.2)

LL.CA: An Implementation of the Livingston and Lewis (1995) Approach to Estimate Classification Consistency and Accuracy based on Observed Test Scores and Test Reliability.

Description

An implementation of what has been come to be known as the "Livingston and Lewis approach" to classification consistency and accuracy, which by employing a compound beta-binomial distribution assumes that true-scores conform to the four-parameter beta distribution, and errors of measurement to the binomial distribution. Under these assumptions, the expected classification consistency and accuracy of tests can be estimated from observed outcomes and test reliability.

Usage

LL.CA(
  x = NULL,
  reliability,
  cut,
  min = 0,
  max = 1,
  error.model = "binomial",
  truecut = NULL,
  output = c("accuracy", "consistency"),
  override = FALSE
)

Arguments

x

A vector of observed scores for which a beta-distribution is to be fitted, or a list of pre-defined true-score distribution parameter values. If a list is provided, the list entries must be named after the parameters: l and u for the location parameters, and alpha and beta for the shape parameters.

reliability

The observed-score squared correlation (i.e., proportion of shared variance) with the true-score.

cut

The cutoff value for classifying observations into pass or fail categories.

min

The minimum value possible to attain on the test. Default is 0 (assuming x represent proportions).

max

The maximum value possible to attain on the test. Default is 1 (assuming x represent proportions).

error.model

The probability distribution to be used for producing the sampling distributions at different points of the true-score scale. Options are binomial and beta. The binomial distribution is discrete, and is the distribution used originally by Livingston and Lewis. Use of the binomial distribution involves a rounding of the effective test length to the nearest integer value. The Beta distribution is continuous, and does not involve rounding of the effective test length.

truecut

Optional specification of a "true" cutoff. Useful for producing ROC curves.

output

Character vector indicating which types of statistics (i.e, accuracy and/or consistency) are to be computed and included in the output. Permissible values are "accuracy" and "consistency".

override

Logical value indicating whether to override the automatic default to the two-parameter Beta true-score distribution if the four-parameter fitting procedure produces impermissible parameter estimates. Default is FALSE.

Value

A list containing the estimated parameters necessary for the approach (i.e., the effective test-length and the beta distribution parameters), the confusion matrix containing estimated proportions of true/false pass/fail categorizations for a test, diagnostic performance statistics, and / or a classification consistency matrix and indices. Accuracy output includes a confusion matrix and diagnostic performance indices, and consistency output includes a consistency matrix and consistency indices p (expected proportion of agreement between two independent test administrations), p_c (proportion of agreement on two independent administrations expected by chance alone), and Kappa (Cohen's Kappa).

References

Livingston, Samuel A. and Lewis, Charles. (1995). Estimating the Consistency and Accuracy of Classifications Based on Test Scores. Journal of Educational Measurement, 32(2).

Hanson, Bradley A. (1991). Method of Moments Estimates for the Four-Parameter Beta Compound Binomial Model and the Calculation of Classification Consistency Indexes. American College Testing.

Examples

Run this code
# NOT RUN {
# Generate some fictional data. Say, 100 individuals take a test with a
# maximum score of 100 and a minimum score of 0.
set.seed(1234)
testdata <- rbinom(100, 100, rBeta.4P(100, .25, .75, 5, 3))
hist(testdata, xlim = c(0, 100))

# Suppose the cutoff value for attaining a pass is 50 items correct, and
# that the reliability of this test was estimated to 0.7. To estimate and
# retrieve the estimated parameters, confusion matrix, consistency and
# accuracy statistics using LL.CA():
LL.CA(x = testdata, reliability = .7, cut = 50, min = 0, max = 100)

# Alternatively to supplying scores to which a true-score distribution is
# to be fit, a list with true-score distribution parameter values can be
# supplied manually, foregoing the need for actual data. The list entries
# must be named. "l" is the lower-bound and "u" the upper-bound location
# parameters of the true-score distribution, and "alpha" and "beta" the
# shape parameters.
trueparams <- list("l" = 0.25, "u" = 0.75, "alpha" = 5, "beta" = 3)
LL.CA(x = trueparams, reliability = .7, cut = 50, min = 0, max = 100)
# }

Run the code above in your browser using DataCamp Workspace