localLLM (version 1.1.0)

intercoder_reliability: Intercoder reliability for LLM annotations

Description

Intercoder reliability for LLM annotations

Usage

intercoder_reliability(
  annotations,
  method = c("auto", "cohen", "krippendorff"),
  label_levels = NULL,
  sample_col = "sample_id",
  model_col = "model_id",
  label_col = "label"
)

Value

A list containing `cohen` (data frame of pairwise kappas) and/or `krippendorff` (overall alpha statistic with per-item agreement scores).

Arguments

annotations

Output from [explore()] or a compatible data frame with at least `sample_id`, `model_id`, and `label` columns.

method

One of `"auto"`, `"cohen"`, or `"krippendorff"`. The `"auto"` setting computes both pairwise Cohen's Kappa and Krippendorff's Alpha (when applicable).

label_levels

Optional factor levels to enforce a consistent ordering in the resulting tables.

sample_col

Column name that identifies samples when `annotations` is a user-provided data frame.

model_col

Column name for the model identifier when using a custom `annotations` data frame.

label_col

Column name containing model predictions when using a custom `annotations` data frame.