reproducer (version 0.7.0)

wmcc_sensitivity: wmcc_sensitivity

Description

Compares two classifiers across a range of FN:FP cost ratios (w) to assess whether their Weighted Matthews Correlation Coefficient (compute_wmcc) ranking is robust or flips at some crossover point, as used in the sensitivity analyses of Madeyski, Kitchenham and Shepperd (2026). For each value of w the WMCC of both classifiers is computed, the winner is recorded, and (as a side effect) a short summary identifying the approximate crossover w is printed.

Usage

wmcc_sensitivity(
  TP_A,
  FP_A,
  FN_A,
  TN_A,
  TP_B,
  FP_B,
  FN_B,
  TN_B,
  label_A = "Model A",
  label_B = "Model B",
  w_range = 1:20
)

Value

a data frame with columns w, the WMCC of each classifier (named after label_A and label_B), Winner and Gap (WMCC_A minus WMCC_B).

Arguments

TP_A, FP_A, FN_A, TN_A

the confusion matrix counts of classifier A.

TP_B, FP_B, FN_B, TN_B

the confusion matrix counts of classifier B.

label_A

a display name for classifier A (default "Model A").

label_B

a display name for classifier B (default "Model B").

w_range

a vector of w values to evaluate (default 1:20).

Author

Lech Madeyski

References

Lech Madeyski, Barbara Kitchenham, Martin Shepperd (2026). LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews. Information and Software Technology, 198, 108204. doi:10.1016/j.infsof.2026.108204.

Examples

# Syriani et al. RL4SE: ChatGPT vs LR (paper Table 4); ranking flips around w = 6-7
wmcc_sensitivity(
  77, 310, 17, 685,  # ChatGPT
  56, 130, 38, 865,  # LR
  label_A = "ChatGPT", label_B = "LR")