E2E (version 0.1.2)

evaluate_model_dia: Evaluate Diagnostic Model Performance

Description

Evaluates the performance of a trained diagnostic model using various metrics relevant to binary classification, including AUROC, AUPRC, and metrics at an optimal or specified probability threshold.

Usage

evaluate_model_dia(
  model_obj = NULL,
  X_data = NULL,
  y_data,
  sample_ids,
  threshold_choices = "default",
  pos_class,
  neg_class,
  precomputed_prob = NULL,
  y_original_numeric = NULL
)

Value

A list containing:

• sample_score: A data frame with sample (ID), label (original numeric), and score (predicted probability for positive class).

• evaluation_metrics: A list of performance metrics:

  • Threshold_Strategy: The strategy used for threshold selection.

  • _Threshold: The chosen probability threshold.

  • Accuracy, Precision, Recall, F1, Specificity: Metrics calculated at _Threshold.

  • AUROC: Area Under the Receiver Operating Characteristic curve.

  • AUROC_95CI_Lower, AUROC_95CI_Upper: 95% confidence interval for AUROC.

  • AUPRC: Area Under the Precision-Recall curve.

Arguments

model_obj

A trained model object (typically a caret::train object or a list from an ensemble like Bagging). Can be NULL if precomputed_prob is provided.

X_data

A data frame of features corresponding to the data used for evaluation. Required if model_obj is provided and precomputed_prob is NULL.

y_data

A factor vector of true class labels for the evaluation data.

sample_ids

A vector of sample IDs for the evaluation data.

threshold_choices

A character string specifying the thresholding strategy ("default", "f1", "youden") or a numeric probability threshold value (0-1).

pos_class

A character string, the label for the positive class.

neg_class

A character string, the label for the negative class.

precomputed_prob

Optional. A numeric vector of precomputed probabilities for the positive class. If provided, model_obj and X_data are not used for score derivation.

y_original_numeric

Optional. The original numeric/character vector of labels. If not provided, it's inferred from y_data using global pos_label_value and neg_label_value.

Examples

# \donttest{
set.seed(42)
n_obs <- 50
X_toy <- data.frame(
  FeatureA = rnorm(n_obs),
  FeatureB = runif(n_obs, 0, 100)
)
y_toy <- factor(sample(c("Control", "Case"), n_obs, replace = TRUE),
                levels = c("Control", "Case"))
ids_toy <- paste0("Sample", 1:n_obs)

# 2. Train a model
rf_model <- rf_dia(X_toy, y_toy)

# 3. Evaluate the model using F1-score optimal threshold
eval_results <- evaluate_model_dia(
  model_obj = rf_model,
  X_data = X_toy,
  y_data = y_toy,
  sample_ids = ids_toy,
  threshold_choices = "f1",
  pos_class = "Case",
  neg_class = "Control"
)
str(eval_results)
# }