Computes weighted Cohen's kappa for either a pair of ordinal rating vectors or all pairwise combinations of ordinal columns in a matrix or data frame.
weighted_kappa(
data,
y = NULL,
weights = c("quadratic", "linear", "unweighted"),
levels = NULL,
na_method = c("error", "pairwise", "complete"),
ci = FALSE,
p_value = FALSE,
conf_level = 0.95,
n_threads = getOption("matrixCorr.threads", 1L),
output = c("matrix", "sparse", "edge_list"),
threshold = 0,
diag = TRUE,
...
)# S3 method for weighted_kappa
print(
x,
digits = 4,
n = NULL,
topn = NULL,
max_vars = NULL,
width = NULL,
show_ci = NULL,
...
)
If y is supplied, a scalar S3 object of class
c("weighted_kappa", "numeric") is returned with diagnostics and
optional ci and inference attributes. Otherwise a symmetric
matrix-style result with estimator class weighted_kappa.
In matrix mode, a matrix or data frame whose rows are observational units and whose columns are raters or classifiers. Each column contains ordered category ratings. In two-vector mode, the first ordinal rating vector.
Optional second ordinal rating vector. When supplied, the function
returns a single weighted kappa estimate for data and y.
Weight specification. The default "quadratic" uses
Fleiss-Cohen-style agreement weights. "linear" uses equal-spacing
agreement weights. "unweighted" reduces to ordinary Cohen's kappa.
Custom numeric weight matrices must be symmetric agreement weights with
diagonal 1 and entries in [0, 1].
Optional ordered category levels. Weighted kappa depends on category order and will not silently alphabetise arbitrary labels. If omitted, order is inferred only when all involved ratings are ordered factors with identical levels or when all involved ratings are numeric or integer and can be ordered by their sorted unique values.
Character scalar controlling missing-data handling.
"error" rejects missing values. "complete" applies listwise
deletion across retained columns before computing the matrix. In matrix
mode, "pairwise" uses pair-specific complete observations.
Logical; if TRUE, attach large-sample delta-method
confidence intervals.
Logical; if TRUE, attach large-sample delta-method
Wald test statistics and p-values for \(H_0: \kappa_w = 0\).
Confidence level used when ci = TRUE. Default is
0.95.
Integer \(\ge 1\). Number of OpenMP threads used by the
matrix C++ backend. Defaults to
getOption("matrixCorr.threads", 1L).
Output representation for matrix mode.
"matrix" (default): full dense matrix.
"sparse": sparse matrix from Matrix containing only
retained entries after thresholding.
"edge_list": long-form data frame representation.
Non-negative absolute-value filter for non-matrix outputs:
retain entries with abs(value) >= threshold. Must be 0 when
output = "matrix".
Logical; whether to include diagonal entries in sparse and edge-list outputs.
Reserved for future extensions. Unsupported extra arguments are rejected.
A matrix-style weighted_kappa object.
Integer; number of decimal places for displayed values.
Optional preview row threshold.
Optional number of leading/trailing rows to show when truncated.
Optional maximum number of visible columns.
Optional display width.
One of "yes" or "no".
Thiago de Paula Oliveira
This implementation uses agreement/similarity weights internally:
"unweighted": \(w_{ij} = 1(i = j)\)
"linear": \(w_{ij} = 1 - |i - j| / (K - 1)\)
"quadratic": \(w_{ij} = 1 - (i - j)^2 / (K - 1)^2\)
The "quadratic" scheme is the default and corresponds to
Fleiss-Cohen-style agreement weights. The "linear" scheme
corresponds to equal-spacing agreement weights.
For matrix mode, rows are shared observational units and columns are raters
or classifiers. A common category map is resolved in R and the main
computation is performed in C++. Missing-data handling follows the usual
matrixCorr na_method conventions. Pairwise complete counts are
stored in attr(x, "diagnostics")$n_complete.
Confidence intervals and standard errors.
Confidence intervals and p-values use the exact large-sample multinomial
delta-method formula. Let
\(A_w = \sum_{a,b} w_{ab} p_{ab}\) be the observed weighted agreement,
\(E_w = \sum_{a,b} w_{ab} q_a r_b\) the expected weighted agreement,
and \(D = 1 - E_w\). Define the weighted margin summaries
$$ u_a = \sum_b w_{ab} r_b, \qquad v_b = \sum_a w_{ab} q_a. $$
For each cell \((a,b)\), the backend uses
$$ g_{ab}
\;=\;
\frac{D\,w_{ab} + (A_w - 1)\,(u_a + v_b)}{D^2}. $$
The variance estimator is
$$ \widehat{\mathrm{Var}}(\hat\kappa_w)
\;=\;
\frac{1}{n}\left(
\sum_{a,b} p_{ab} g_{ab}^2 -
\Big(\sum_{a,b} p_{ab} g_{ab}\Big)^2
\right), $$
where \(n\) is the number of complete paired ratings. The standard error
is \(\widehat{\mathrm{se}}(\hat\kappa_w)=
\sqrt{\widehat{\mathrm{Var}}(\hat\kappa_w)}\) and the CI is the Wald interval
$$ \hat\kappa_w \pm z_{1-\alpha/2}\,\widehat{\mathrm{se}}(\hat\kappa_w), $$
truncated to \([-1, 1]\) in the returned result. Use cohen_kappa() for
unordered nominal categories where all disagreements are equally serious.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213-220. tools:::Rd_expr_doi("10.1037/h0026256")
cohen_kappa() for unweighted two-rater nominal agreement;
multirater_kappa() for panel-level nominal agreement among three or more
raters.
raters <- data.frame(
r1 = ordered(c("low", "low", "mid", "high", "high"),
levels = c("low", "mid", "high")),
r2 = ordered(c("low", "mid", "mid", "high", "high"),
levels = c("low", "mid", "high")),
r3 = ordered(c("low", "low", "high", "high", "mid"),
levels = c("low", "mid", "high"))
)
wk <- weighted_kappa(raters)
print(wk)
summary(wk)
estimate(wk)
tidy(wk)
plot(wk)
x <- raters$r1
y <- raters$r2
weighted_kappa(x, y, weights = "linear")
Run the code above in your browser using DataLab