icr (version 0.5.2)

krippalpha: Krippendorff's alpha

Description

krippalpha computes Krippendorff's reliability coefficient alpha.

Usage

krippalpha(data, metric = "nominal", bootstrap = FALSE, bootnp = FALSE,
  nboot = 20000, nnp = 1000, cores = 1, custom_seed = NULL)

Arguments

data

a matrix of reliability data.

metric

metric difference function to be applied to disagreements. Supports nominal, ordinal, interval, and ratio. Defaults to nominal.

bootstrap

logical indicating whether uncertainty estimates should be obtained using the bootstrap algorithm defined by Krippendorff. Defaults to FALSE.

bootnp

logical indicating whether non-parametric bootstrap uncertainty estimates should be computed. Defaults to FALSE.

nboot

number of bootstraps used in Krippendorff's algorithm. Defaults to 20000.

nnp

number of non-parametric bootstraps. Defaults to 1000.

cores

number of cores across which bootstrap-computations are distributed. Defaults to 1. If more cores are specified than available, the number will be set to the maximum number of available cores.

custom_seed

numeric vector of length 6 for the internal L'Ecuyer-CMRG random number generator (see details). Defaults to NULL. When set to NULL, relies on R's .Random.seed vector.

Value

Returns a list of type icr with following elements:

alpha

value of inter-coder reliability coefficient

method

data level of x

n_coders

number of coders

n_units

number of units to be coded

n_values

number of unique values in reliability data

coincidence_matrix

matrix containing coincidences within coder-value pairs

delta_matrix

matrix of metric differences depending on method

D_e

expected disagreement

D_o

observed disagreement

bootstrap

TRUE if Krippendorff bootstrapping algorithm was run, FALSE otherwise

nboot

number of bootstraps

bootnp

TRUE if nonparametric bootstrap was run, FALSE otherwise

nnp

number of non-parametric bootstraps

bootstraps

vector of bootstrapped values of alpha (Krippendorff's algorithm)

bootstrapsNP

vector of non-parametrically bootstrapped values of alpha

Details

For proper seeding of krippalpha's bootstrap-routines via R, specify set.seed(seed, kind = "L'Ecuyer-CMRG"). The seeds returned from R in .Random.seed are internally regarded as 32-bit unsigned integers (see Random), yet represented as 32-bit signed integers. krippalpha will hence convert the seed values obtained via set.seed and user-provided custom_seeds to unsigned 32-bit integers.

Please note that krippalpha takes the seed vector to seed the internal random number generator of both bootstrap-routines. Furthermore, it does not advance R's RNG state. .Random.seed will therefore be the same after krippalpha has been run.

References

Krippendorff, K. (2004) Content Analysis: An Introduction to Its Methodology. Beverly Hills: Sage.

Krippendorff, K. (2011) Computing Krippendorff's Alpha Reliability. Departmental Papers (ASC) 43. http://repository.upenn.edu/asc_papers/43.

Krippendorff, K. (2016) Bootstrapping Distributions for Krippendorff's Alpha. http://web.asc.upenn.edu/usr/krippendorff/boot.c-Alpha.pdf.

L'Ecuyer, P. (1999) Good Parameter Sets for Combined Multiple Recursive Random Number Generators. Operations Research, 47 (1), 159--164. https://pubsonline.informs.org/doi/10.1287/opre.47.1.159.

L'Ecuyer, P., Simard, R, Chen, E. J., and Kelton, W. D. (2002) An Objected-Oriented Random-Number Package with Many Long Streams and Substreams. Operations Research, 50 (6), 1073--1075. http://www.iro.umontreal.ca/~lecuyer/myftp/streams00/c++/streams4.pdf.

Examples

# NOT RUN {
data(codings)
krippalpha(codings)

set.seed(100, kind = "L'Ecuyer-CMRG")
krippalpha(codings, metric = "nominal", bootstrap = TRUE, bootnp = TRUE)

# }