reproducer (version 0.7.0)

llm4screenlit_metrics: llm4screenlit_metrics

Description

Computes all of the metrics recommended by Madeyski, Kitchenham and Shepperd (2026) for evaluating large language models used to screen literature in systematic reviews, directly from confusion matrix counts: the sample size N, Recall, LostEvidence (\(1 - Recall\), i.e. the proportion of relevant studies missed), the standard MCC (compute_mcc) and the weighted WMCC (compute_wmcc). The inputs may be scalars or equal-length vectors, enabling batch computation across many configurations.

Usage

llm4screenlit_metrics(TP, FP, FN, TN, w = 10)

Value

a data frame with columns TP, FP, FN, TN, N, Recall, LostEvidence, MCC and WMCC (the metric columns are rounded to 3 decimal places).

Arguments

TP

the number of true positives (scalar or vector).

FP

the number of false positives (scalar or vector).

FN

the number of false negatives (scalar or vector).

TN

the number of true negatives (scalar or vector).

w

the FN:FP cost ratio used for the WMCC (default 10).

Author

Lech Madeyski

References

Lech Madeyski, Barbara Kitchenham, Martin Shepperd (2026). LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews. Information and Software Technology, 198, 108204. doi:10.1016/j.infsof.2026.108204.

Examples

# Single configuration: Felizardo et al. SLR2, Likert >= 4 (paper Table 3)
llm4screenlit_metrics(TP = 128, FP = 68, FN = 20, TN = 232, w = 10)

# Batch computation: all four Felizardo configurations
llm4screenlit_metrics(
  TP = c(50, 48, 128, 113),
  FP = c(35, 17, 68, 27),
  FN = c(14, 16, 20, 35),
  TN = c(35, 53, 232, 273),
  w = 10)