Estimates Krippendorff's alpha as a panel-level reliability/agreement
coefficient among two or more coders, raters, observers, judges, or
instruments. Missing ratings are supported through na_method, and
nominal, ordinal, interval, and ratio disagreement functions are available.
krippendorff_alpha(
data,
levels = NULL,
input = c("ratings", "counts"),
level = c("nominal", "ordinal", "interval", "ratio"),
method = c("customary", "analytical"),
na_method = c("error", "complete", "available"),
min_raters = 2L,
ci = FALSE,
p_value = FALSE,
conf_level = 0.95,
se_method = c("auto", "bootstrap", "jackknife", "none"),
n_boot = 1000L,
seed = NULL,
n_threads = getOption("matrixCorr.threads", 1L),
return_matrices = FALSE,
verbose = FALSE,
...
)# S3 method for krippendorff_alpha
print(x, digits = 4, ...)
# S3 method for krippendorff_alpha
summary(object, digits = 4, ...)
# S3 method for summary.krippendorff_alpha
print(
x,
digits = NULL,
n = NULL,
topn = NULL,
max_vars = NULL,
width = NULL,
show_ci = NULL,
...
)
# S3 method for krippendorff_alpha
plot(
x,
type = c("estimate", "item_disagreement", "category_proportion", "coincidence"),
...
)
A one-row data frame with class
c("krippendorff_alpha", "agreement_result", "data.frame"). The
object stores method metadata, diagnostics, and optional matrices in
attributes.
Input data.
For input = "ratings", rows are units/items and columns are
coders/raters. For input = "counts", rows are units/items and
columns are categories containing per-item category counts.
Optional category labels in analysis order. For ordinal
character data, explicit levels are required unless the data are
ordered factors. For interval and ratio alpha, levels must be
coercible to finite numeric values.
One of "ratings" or "counts".
Measurement level: "nominal", "ordinal",
"interval", or "ratio".
Estimator. "customary" uses the coincidence-matrix
estimator of Krippendorff/Hayes. "analytical" uses the improved
estimator and jackknife inference direction described by Hughes.
Missing-data rule. "error" rejects missing ratings,
"complete" removes any item with one or more missing ratings, and
"available" retains items with at least min_raters observed
ratings. For input = "counts", missing counts are not allowed and
rows with total counts below min_raters are dropped.
Minimum number of observed ratings required for a retained
item. Must be at least 2.
Logical; if TRUE, attach confidence intervals.
Logical; if TRUE, attach test statistics and p-values.
This is available only for method = "analytical" with jackknife
inference.
Confidence level for intervals. Default is 0.95.
Standard-error method. "auto" resolves to
"bootstrap" for customary alpha when ci = TRUE, and to
"jackknife" for analytical alpha when ci = TRUE or
p_value = TRUE. "bootstrap" is allowed only for customary
alpha. "jackknife" is the primary inference path for analytical
alpha. "none" disables uncertainty output.
Number of bootstrap resamples used for customary alpha when
ci = TRUE.
Optional positive integer seed used for reproducible bootstrap resampling.
Integer \(\ge 1\). Number of OpenMP threads used by the C++ backend.
Logical; if TRUE, attach the retained count
matrix, observed coincidence matrix, expected coincidence matrix, and
disagreement matrix in attr(x, "matrices").
Logical; if TRUE, emit short progress messages.
Unused.
A krippendorff_alpha object.
Integer; number of decimal places for rounded numeric columns.
A krippendorff_alpha object.
Optional preview row threshold.
Optional number of leading/trailing rows to show when truncated.
Optional maximum number of visible columns.
Optional display width.
One of "yes" or "no".
Plot type: "estimate", "item_disagreement",
"category_proportion", or "coincidence".
Thiago de Paula Oliveira
Krippendorff's alpha is a panel-level reliability coefficient, not a pairwise correlation matrix. The customary coincidence-matrix estimator is $$ \alpha = 1 - \frac{D_o}{D_e}, $$ where \(D_o\) is the observed disagreement and \(D_e\) is the expected disagreement under chance assignment.
For retained item \(u\) with \(m_u\) observed ratings and category counts \(n_{uc}\), the observed coincidence matrix updates by $$ O_{ck} \leftarrow O_{ck} + \frac{n_{uc}(n_{uk} - 1(c=k))}{m_u - 1}. $$ Let \(n_c = \sum_k O_{ck}\) and \(n = \sum_c n_c\). Then $$ D_o = \frac{1}{n}\sum_{c,k} O_{ck}\,\delta^2_{ck}, $$ $$ D_e = \frac{1}{n(n-1)}\sum_{c,k} n_c n_k\,\delta^2_{ck}, $$ where \(\delta^2_{ck}\) is the level-specific disagreement matrix.
The disagreement functions implemented here are:
nominal: \(\delta^2_{ck} = 1(c \ne k)\)
ordinal: Krippendorff's cumulative-margin disagreement based on the pooled category margins
interval: \(\delta^2_{ck} = (v_c - v_k)^2\)
ratio: \(\delta^2_{ck} = (v_c - v_k)^2 / (v_c + v_k)^2\), with \(\delta^2_{ck}=0\) when both values are zero
Analytical estimator and inference. Hughes (2024) recommends a different point estimator based on within-item and pooled pairwise disagreement. Let \(a\) be the number of retained items, \(m_i\) the number of ratings in item \(i\), and \(N = \sum_i m_i\). Define $$ \bar n = \frac{N - \sum_i m_i^2 / N}{a - 1}. $$ The within-item mean square is $$ MSE = \frac{1}{N}\sum_{i=1}^a \frac{\sum_{j<k}\delta^2(x_{ij}, x_{ik})}{m_i - 1}, $$ the pooled total disagreement is $$ SST = \frac{1}{N}\sum_{r<s}\delta^2(z_r, z_s), $$ where \(z_1,\ldots,z_N\) are the pooled observed ratings, and $$ SSA = SST - (N-a)\,MSE, \qquad MSA = \frac{SSA}{a-1}. $$ With \(\theta = MSA/MSE\) and \(\eta = \log(\theta)\), the analytical alpha estimate is $$ \alpha = \frac{\theta - 1}{\theta + \bar n - 1}. $$
When analytical inference is requested, the implementation uses the delete-one-item jackknife on \(\eta\). If \(\eta_{(-i)}\) is the leave-one-item-out transform and \(a\) is the number of retained items, the jackknife pseudo-values are $$ p_i = a\,\hat\eta - (a-1)\,\hat\eta_{(-i)}, $$ with standard error $$ \widehat{\mathrm{se}}(\hat\eta) = \sqrt{\mathrm{Var}(p_1,\ldots,p_a) / a}. $$ The confidence interval is formed on the \(\eta\)-scale with a \(t_{a-1}\) critical value and back-transformed to alpha using the full data \(\bar n\). The analytical p-value tests \(H_0: \alpha = 0\), equivalently \(H_0: \eta = 0\), via the t statistic $$ t = \hat\eta / \widehat{\mathrm{se}}(\hat\eta). $$
For customary alpha, percentile bootstrap intervals are available by resampling retained items with replacement and recomputing the customary estimate.
Krippendorff, K. (2011/2013). Computing Krippendorff's alpha-reliability.
Hayes, A. F. and Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89.
Hughes, J. (2024). Toward improved inference for Krippendorff's Alpha agreement coefficient. Journal of Statistical Planning and Inference, 233, 106170.
multirater_kappa() for nominal multi-rater kappa;
gwet_ac() for AC1/AC2 agreement coefficients.
raters <- data.frame(
r1 = c("A", "A", "B", "B", "C", "A"),
r2 = c("A", "B", "B", "B", "C", "A"),
r3 = c("A", "A", "B", "C", "C", "A"),
stringsAsFactors = FALSE
)
fit <- krippendorff_alpha(raters, level = "nominal", na_method = "available")
print(fit)
summary(fit)
estimate(fit)
tidy(fit)
plot(fit)
Run the code above in your browser using DataLab