Estimates Gwet's chance-corrected agreement coefficient for either unweighted nominal agreement (AC1) or weighted agreement (AC2) in two-rater pairwise form or panel-level multi-rater form.
gwet_ac(
data,
y = NULL,
weights = c("unweighted", "linear", "quadratic", "ordinal", "radical", "ratio",
"circular", "bipolar"),
levels = NULL,
input = c("pairwise", "ratings", "counts"),
na_method = c("error", "pairwise", "complete", "available"),
min_raters = 2L,
by_category = FALSE,
ci = FALSE,
p_value = FALSE,
conf_level = 0.95,
se_method = c("asymptotic", "jackknife", "none"),
n_threads = getOption("matrixCorr.threads", 1L),
output = c("matrix", "sparse", "edge_list"),
threshold = 0,
diag = TRUE,
verbose = FALSE,
...
)# S3 method for gwet_ac
summary(object, digits = 4, ci_digits = 3, p_digits = 4, ...)
# S3 method for summary.gwet_ac
print(
x,
digits = NULL,
n = NULL,
topn = NULL,
max_vars = NULL,
width = NULL,
show_ci = NULL,
...
)
# S3 method for gwet_ac
print(
x,
digits = 4,
n = NULL,
topn = NULL,
max_vars = NULL,
width = NULL,
show_ci = NULL,
show_by_category = FALSE,
...
)
# S3 method for gwet_ac
plot(
x,
title = NULL,
low_color = "indianred1",
high_color = "steelblue1",
mid_color = "white",
value_text_size = 4,
ci_text_size = 3,
show_value = TRUE,
type = c("agreement_map", "estimate", "item_agreement", "category_proportion",
"by_category"),
bins = 30L,
...
)
If y is supplied, a scalar numeric object of class
c("gwet_ac", "numeric"). If input = "pairwise", a
corr_result. If input is "ratings" or "counts",
a one-row data frame with class
c("gwet_ac", "agreement_result", "data.frame").
Input data. If y is supplied, data is the first
rater vector. Otherwise:
for input = "pairwise", rows are units and columns are raters;
for input = "ratings", rows are items and columns are raters;
for input = "counts", rows are items and columns are categories
containing rater counts.
Optional second rater vector for scalar two-rater agreement.
Weight specification. "unweighted" yields AC1.
Weighted schemes yield AC2. Supported built-in schemes are
"linear", "quadratic", "ordinal",
"radical", "ratio", "circular", and
"bipolar". A custom numeric agreement-weight matrix may also be
supplied. Character schemes must be supplied as a single value.
Optional explicit category labels. This is mainly useful when unused categories should still be retained in the calculation, and is the recommended way to define ordering for ordinal AC2 weights.
One of "pairwise", "ratings", or "counts".
Missing-data rule. For scalar two-rater and pairwise matrix
modes, "error" rejects missing values, "complete" applies
listwise deletion in matrix mode, and "pairwise" uses complete
pairs within each pair of raters. For input = "ratings",
"available" retains items with at least min_raters
observed ratings. For input = "counts", missing values are not
allowed.
Minimum number of observed ratings required for an item to
be retained in panel mode. Must be at least 2.
Logical; if TRUE, attach one-vs-all category-wise
AC1 estimates for panel-level fits. This is available only for the
unweighted coefficient.
Logical; if TRUE, attach confidence intervals.
Logical; if TRUE, attach test statistics and p-values.
Confidence level for intervals. Default is 0.95.
Standard-error method. "asymptotic" uses the
analytic large-sample formula. "jackknife" is retained as an
alternative for panel-level fits. "none" disables inferential
output.
Integer \(\ge 1\). Number of OpenMP threads used by the C++ backend.
One of "matrix", "sparse", or
"edge_list" for pairwise matrix mode.
Numeric threshold used only for thresholded pairwise matrix output.
Logical; include diagonal entries in thresholded pairwise matrix output.
Logical; if TRUE, emit short progress messages.
Unused.
A gwet_ac object.
Integer; number of decimal places for displayed values.
Integer; number of decimal places for confidence limits.
Integer; number of decimal places for p-values.
A gwet_ac object.
Optional preview row threshold.
Optional number of leading/trailing rows to show when truncated.
Optional maximum number of visible columns.
Optional display width.
One of "yes" or "no".
Logical; whether to print attached category-wise panel results when available.
Optional plot title.
Fill/color used for negative agreement.
Fill/color used for positive agreement.
Unused placeholder for API consistency with matrix heatmaps.
Text size for estimate labels in matrix plots.
Text size for CI labels in matrix plots.
Logical; whether to print estimate and CI labels.
Plot type for panel-level fits: "agreement_map",
"estimate", "item_agreement", "category_proportion",
or "by_category".
Integer number of bins retained for compatibility with the panel-level item-agreement plot.
Thiago de Paula Oliveira
For two raters, let \(w_{ab}\) denote the agreement weight for cell \((a,b)\), \(p_{ab}\) the cell proportion, and \(\pi_k = (p_{k+} + p_{+k})/2\). Then $$ P_o = \sum_{a,b} w_{ab} p_{ab}, $$ $$ P_e = \frac{\sum_{a,b} w_{ab}}{q(q-1)} \sum_{k=1}^q \pi_k(1-\pi_k), $$ and $$ AC = \frac{P_o - P_e}{1 - P_e}. $$ With identity weights \(w_{ab} = 1(a=b)\) this is Gwet's AC1, the nominal-agreement coefficient. With non-identity agreement weights this is Gwet's AC2, which extends the same chance-correction idea to ordered or partially creditable disagreement by letting near disagreements receive larger agreement weights than distant disagreements.
Category ordering for weighted AC2. Built-in weighted schemes use
the category order to define near and distant disagreements. If
levels is supplied, that order is used. Otherwise, factor inputs use
their factor levels, numeric/integer/logical inputs use sorted observed
values, character inputs use first-observed order, and counts inputs use
column order (or levels, when supplied). For ordinal analyses,
supplying levels is recommended whenever first-observed character
order is not the intended scale order.
For panel-level counts, if \(r_{ik}\) is the number of raters assigning item \(i\) to category \(k\), \(r_i = \sum_k r_{ik}\), and \(w_{kl}\) is the agreement-weight matrix, then the item-level agreement is $$ A_i = \frac{1}{r_i(r_i-1)} \sum_k r_{ik}\left(\sum_l w_{kl}r_{il}-1\right). $$ The observed agreement is the mean of \(A_i\) over retained items. The chance term uses the average item-wise category proportions \(\pi_k = m^{-1}\sum_i r_{ik}/r_i\), where \(m\) is the number of retained items.
Confidence intervals and inference.
The primary inferential path is the analytic large-sample method. For two raters, define $$ s_{ab} = w_{ab} - \frac{2(1-AC)\sum_{u,v} w_{uv}}{q(q-1)} \left(1 - \frac{\pi_a + \pi_b}{2}\right). $$ The backend variance estimator is $$ \widehat{\mathrm{Var}}(\widehat{AC}) = \frac{1}{n(1-P_e)^2} \left[ \sum_{a,b} p_{ab} s_{ab}^2 - \left\{P_o - 2(1-AC)P_e\right\}^2 \right], $$ where \(n\) is the number of complete paired ratings.
For panel-level counts, write $$ AC_i = \frac{A_i - P_e}{1 - P_e}, $$ and define the item-specific chance term $$ P_{e,i} = \frac{\sum_{k,l} w_{kl}}{q(q-1)} \sum_{k=1}^q \frac{r_{ik}}{r_i}(1-\pi_k). $$ The asymptotic linearised contribution used by the backend is $$ \xi_i = AC_i - \frac{2(1-AC)}{1-P_e}(P_{e,i} - P_e), $$ giving $$ \widehat{\mathrm{Var}}(\widehat{AC}) = \frac{1}{m(m-1)} \sum_{i=1}^m (\xi_i - AC)^2, $$ where \(m\) is the number of retained items.
The reported standard error is
\(\widehat{\mathrm{se}}(\widehat{AC}) =
\sqrt{\widehat{\mathrm{Var}}(\widehat{AC})}\), and the confidence interval is
the t interval
$$ \widehat{AC} \pm t_{1-\alpha/2,\nu}\,
\widehat{\mathrm{se}}(\widehat{AC}), $$
truncated to \([-1,1]\). Here \(\nu = n-1\) for the two-rater path and
\(\nu = m-1\) for panel-level asymptotic inference. The reported test
statistic is the corresponding t ratio for \(H_0: AC = 0\). For
panel-level fits, se_method = "jackknife" remains available as a
second option, using the leave-one-item-out variance
$$ \widehat{\mathrm{Var}}_{\mathrm{JK}}(\widehat{AC})
=
\frac{m-1}{m}
\sum_{i=1}^m
\left(\widehat{AC}_{(-i)} - \overline{AC}_{(-\cdot)}\right)^2. $$
Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61, 29-48. tools:::Rd_expr_doi("10.1348/000711006X126600")
cohen_kappa() for two-rater nominal kappa;
multirater_kappa() for panel-level nominal kappa;
weighted_kappa() for weighted Cohen-type agreement.
x <- c("A", "A", "B", "B", "A", "C")
y <- c("A", "B", "B", "B", "A", "C")
gwet_ac(x, y)
gwet_ac(x, y, weights = "quadratic", levels = c("A", "B", "C"))
raters <- data.frame(
r1 = c("A", "A", "B", "C", "A", "B"),
r2 = c("A", "B", "B", "C", "A", "B"),
r3 = c("A", "A", "B", "B", "A", "C"),
stringsAsFactors = FALSE
)
fit_pw <- gwet_ac(raters)
fit_panel <- gwet_ac(raters, input = "ratings")
print(fit_pw)
print(fit_panel)
estimate(fit_pw)
tidy(fit_pw)
tidy(fit_panel)
Run the code above in your browser using DataLab