rtemis (version 1.4.3)

calibrate: Calibrate Classification & ClassificationRes Models

Description

Generic function to calibrate binary classification models.

Usage

calibrate(
  x,
  hyperparameters = NULL,
  execution_config = NULL,
  verbosity = 1L,
  ...
)

Value

Calibrated model object.

Arguments

x

Classification or ClassificationRes object to calibrate.

hyperparameters

Optional Hyperparameters object: Setup using one of setup_* functions. Defines the algorithm used to train the calibration model. NULL uses setup_Isotonic.

execution_config

Optional ExecutionConfig object: Setup using setup_SerialExecution, setup_FutureExecution or setup_MiraiExecution. The calibration models are trained under it. NULL trains them serially.

verbosity

Integer: Verbosity level.

...

Additional arguments passed to specific methods.

Method-specific parameters

For Classification objects:

• predicted_probabilities: Numeric vector of the positive class's predicted probabilities, one per case

• true_labels: Factor of true class labels

For ClassificationRes objects:

• resampler_config: ResamplerConfig object for calibration training

• train_verbosity: Integer controlling calibration model training output

Choosing a calibrator

A calibration map must be monotonic non-decreasing. A map that reorders scores changes the ranking of the predictions, and so changes AUC; a non-decreasing one cannot. available_calibration lists the algorithms that carry that guarantee. Any other Hyperparameters object is accepted and trained like any other model, and the map is then unconstrained.

• setup_Isotonic (the default) fits isotonic regression. The map is a step function, so it merges nearby scores into ties, which moves AUC slightly in either direction. Fitted probabilities are held at least 1 / (2 * n) away from 0 and 1, n being the number of calibration cases, so a block of uniformly labelled cases does not assert certainty.

• setup_MonotonicHAL fits a monotonic Highly Adaptive Lasso on the logit scale. At its default smoothness_orders = 1 the map is continuous and strictly increasing, so it preserves AUC exactly -- but the constraint that achieves this also makes the map convex on the logit scale, which costs accuracy whenever the correction needed is concave. smoothness_orders = 0 with penalized = FALSE lifts that restriction, at the cost of ties and a markedly slower fit.

In rtemis benchmarks, no monotonic lasso configuration was uniformly more accurate than isotonic regression, and each was several times slower. data-raw/benchmark_calibrators.R reproduces the comparison.

The calibrator that ran is recorded on the returned object's @calibrator property, is shown by print(), and is serialized by to_json().

Author

EDG

Details

The goal of calibration is to adjust the predicted probabilities of a binary classification model so that they better reflect the true probabilities (i.e. empirical risk) of the positive class.

Examples

# --- Calibrate Classification ---
dat <- iris[51:150, ]
res <- resample(dat)
dat[["Species"]] <- factor(dat[["Species"]])
dat_train <- dat[res[[1]], ]
dat_test <- dat[-res[[1]], ]

# Train GLM on a training/test split
mod_c_glm <- train(
  x = dat_train,
  dat_test = dat_test,
  hyperparameters = setup_GLM()
)

# Calibrate the `Classification` by defining `predicted_probabilities` and `true_labels`,
# in this case using the training data, but it could be a separate calibration dataset.
mod_c_glm_cal <- calibrate(
  mod_c_glm,
  predicted_probabilities = mod_c_glm[["predicted_prob_training"]][, 1L],
  true_labels = mod_c_glm[["y_training"]]
)
mod_c_glm_cal

# --- Calibrate ClassificationRes ---

# Train GLM with cross-validation
resmod_c_glm <- train(
  x = dat,
  hyperparameters = setup_GLM(),
  outer_resampling_config = setup_KFold(n_resamples = 3L)
)

# Calibrate the `ClassificationRes` using the same resampling configuration as used for training.
resmod_c_glm_cal <- calibrate(resmod_c_glm)
resmod_c_glm_cal