Estimates panel-level chance-corrected agreement among multiple raters assigning items to nominal categories.
multirater_kappa(
data,
levels = NULL,
input = c("ratings", "counts"),
method = c("fleiss", "randolph"),
exact = FALSE,
na_method = c("error", "complete", "available"),
min_raters = 2L,
by_category = FALSE,
ci = FALSE,
p_value = FALSE,
conf_level = 0.95,
se_method = c("asymptotic", "jackknife", "none"),
n_threads = getOption("matrixCorr.threads", 1L),
verbose = FALSE,
...
)# S3 method for multirater_kappa
print(x, digits = 4, show_by_category = FALSE, ...)
# S3 method for multirater_kappa
summary(object, digits = 4, ...)
# S3 method for summary.multirater_kappa
print(
x,
digits = NULL,
n = NULL,
topn = NULL,
max_vars = NULL,
width = NULL,
show_ci = NULL,
...
)
# S3 method for multirater_kappa
plot(
x,
type = c("agreement_map", "estimate", "item_agreement", "category_proportion",
"by_category"),
bins = 30L,
...
)
A one-row data frame with class
c("multirater_kappa", "agreement_result", "data.frame"). The object
carries estimator metadata and diagnostics in attributes, and may also
attach category-wise binary-collapsed results in
attr(x, "by_category").
Input ratings or counts data.
For input = "ratings", rows are items/subjects and columns are
raters/classifiers. For input = "counts", rows are items/subjects,
columns are categories, and values are counts of raters assigning that
category to that item.
Optional explicit category labels. For nominal multi-rater
kappa, category order is not used in the estimator itself, but explicit
levels are useful when unobserved categories should be retained in the
output. If factor columns are mixed with non-factor columns, levels
must be supplied to avoid ambiguous implicit ordering.
One of "ratings" or "counts".
One of "fleiss" (fixed-marginal multi-rater kappa) or
"randolph" (free-marginal multi-rater kappa).
Logical; if TRUE, use the exact Fleiss fixed-marginal
variant. This is available only for input = "ratings" with
complete item-by-rater data and method = "fleiss".
Missing-data rule for input = "ratings".
"error" rejects missing ratings. "complete" removes any item
with one or more missing ratings. "available" retains items with at
least min_raters observed ratings and allows the number of raters to
vary by item. For input = "counts", missing counts are not allowed.
Minimum number of observed ratings required for an item to
be retained when na_method = "available" or when
input = "counts" rows are filtered by total raters. Must be at
least 2.
Logical; if TRUE, attach category-wise kappas based
on target-vs-other binary collapses of the category count matrix.
Logical; if TRUE, attach confidence intervals using the
selected se_method.
Logical; if TRUE, attach z statistics and two-sided
p-values using the selected se_method.
Confidence level for intervals. Default is 0.95.
Standard-error method. "asymptotic" uses the
closed-form large-sample variance for the standard non-exact Fleiss kappa
when a common number of raters is available. "jackknife" uses
leave-one-item-out jackknife inference. "none" disables
inferential quantities and may not be combined with ci = TRUE or
p_value = TRUE. When inferential quantities are requested and
se_method is left at its default, multirater_kappa()
automatically falls back to "jackknife" if asymptotic inference is
unavailable for the chosen estimator/data combination. se_method
and conf_level are validated only when ci or p_value
requests inferential output.
Integer \(\ge 1\). Number of OpenMP threads used by the C++ backend.
Logical; if TRUE, emit short progress messages.
Unused.
A multirater_kappa object.
Integer; number of decimal places for displayed values.
Logical; whether to print the attached category-wise kappa table when available.
A multirater_kappa object.
Optional preview row threshold.
Optional number of leading/trailing rows when truncated.
Optional maximum number of visible columns.
Optional display width.
One of "yes" or "no".
Plot type: "agreement_map", "estimate", "item_agreement",
"category_proportion", or "by_category".
Integer number of bins retained for compatibility; currently unused by the default item-agreement profile plot.
Thiago de Paula Oliveira
multirater_kappa() returns one panel-level agreement coefficient, not a
pairwise matrix. Fleiss' kappa is the default fixed-marginal estimator:
$$\kappa = \frac{\bar P - P_e}{1 - P_e},$$
where \(\bar P\) is the mean item-level agreement and
\(P_e = \sum_j p_j^2\) uses the pooled marginal category proportions.
Randolph's free-marginal alternative replaces the expected agreement with
\(P_e = 1 / K\). This function is for nominal categories and does not use
category ordering. Use weighted_kappa() for two-rater ordered-category
agreement and cohen_kappa() for two-rater nominal agreement.
When exact = TRUE, the exact Fleiss fixed-marginal estimate requires
the original item-by-rater rating matrix. In matrixCorr, closed-form
asymptotic inference is only available for the standard non-exact Fleiss
estimator with a common number of raters per item. In other settings, use
se_method = "jackknife" when inferential quantities are needed.
With input = "ratings" and na_method = "available", the
implementation supports unbalanced item-specific numbers of raters as a
generalisation beyond the strict equal-rater Fleiss setting. The returned
diagnostics record the observed per-item rater counts.
Confidence intervals and standard errors. Two inference paths are implemented.
If se_method = "jackknife", the method computes leave-one-item-out
estimates \(\hat\kappa_{(-1)},\ldots,\hat\kappa_{(-m)}\) over the
\(m\) retained items and forms
$$ \bar\kappa_{(.)} = \frac{1}{m}\sum_{i=1}^m \hat\kappa_{(-i)}, $$
$$ \widehat{\mathrm{Var}}_{\mathrm{JK}}(\hat\kappa)
\;=\;
\frac{m-1}{m}\sum_{i=1}^m
\left(\hat\kappa_{(-i)} - \bar\kappa_{(.)}\right)^2. $$
The standard error is
\(\widehat{\mathrm{se}}(\hat\kappa)=
\sqrt{\widehat{\mathrm{Var}}_{\mathrm{JK}}(\hat\kappa)}\) and the CI is
$$ \hat\kappa \pm z_{1-\alpha/2}\,\widehat{\mathrm{se}}(\hat\kappa), $$
truncated to \([-1, 1]\).
If se_method = "asymptotic", the closed-form variance is available
only for the standard non-exact Fleiss estimator
with a common number of raters per item. Let \(m\) be the number of items,
\(n\) the common number of raters, \(p_j\) the pooled category
proportions, and define
$$ S = \sum_j p_j(1-p_j), \qquad
C = \sum_j p_j(1-p_j)(1-2p_j). $$
Then the variance estimator is
$$ \widehat{\mathrm{Var}}(\hat\kappa)
\;=\;
\frac{2\,(S^2 - C)}{S^2\,m\,n\,(n-1)}. $$
The standard error and CI are then obtained from the same Wald form
above and truncated to \([-1, 1]\). When asymptotic inference is
unavailable and inferential output is requested with the default setting,
multirater_kappa() automatically falls back to the jackknife path.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382. tools:::Rd_expr_doi("10.1037/h0031619")
Randolph, J. J. (2005). Free-Marginal Multirater Kappa: An Alternative to Fleiss' Fixed-Marginal Multirater Kappa.
cohen_kappa() for unweighted two-rater nominal agreement;
weighted_kappa() for two-rater ordered-category agreement.
raters <- data.frame(
r1 = c("A", "A", "B", "C", "A", "B"),
r2 = c("A", "B", "B", "C", "A", "B"),
r3 = c("A", "A", "B", "B", "A", "C"),
stringsAsFactors = FALSE
)
fit <- multirater_kappa(raters)
print(fit)
summary(fit)
estimate(fit)
tidy(fit)
# The default plot is an item-by-category agreement map.
# Rows are items, ordered from stronger to weaker item-level agreement.
# Columns are categories. Each tile shows how many raters assigned that
# category to that item, and darker fill means a larger share of raters
# chose that category for that item.
plot(fit)
Run the code above in your browser using DataLab