rtemis (version 1.4.3)

conformal_metrics: Score a Conformal Prediction Region

Description

Empirical coverage and region size for a PredictionRegion, against the outcomes of the cases it bounds.

Usage

conformal_metrics(region, true_outcome)

Value

One-row data.frame. coverage and n for either shape, plus mean_width and median_width for an interval, or mean_set_size, singleton_rate and empty_rate for a set.

Arguments

region

PredictionRegion object, from conformal.

true_outcome

Numeric or Factor: The outcomes of the cases region bounds, in the same order.

Author

EDG

Details

Coverage on the calibration data is not a check. The threshold is chosen as the 1 - alpha empirical quantile of the calibration scores, so coverage measured back on those same cases is 1 - alpha by construction. Coverage is scored against outcomes nothing in the pipeline has seen.

Read coverage and size together. A model with no signal attains valid coverage by returning uninformatively wide intervals, so coverage alone does not distinguish an accurate model from a wide one. Width -- or set size, and the rate of singleton and empty sets -- is the part that carries information about the model.

Coverage is a proportion over the cases supplied, so it carries the Monte Carlo error of that sample: 100 cases at a nominal 90% have a standard error near 3 points, and a single run landing at 87% is not evidence of a broken guarantee.

Examples

x <- data.frame(age = rnorm(300), bmi = rnorm(300))
x[["y"]] <- x[["age"]] * 2 + rnorm(300, sd = 0.3)
mod <- train(
  x[1:200, ],
  dat_test = x[201:250, ],
  hyperparameters = setup_GLM(),
  verbosity = 0L
)
held_out <- x[251:300, ]
region <- conformal(mod, held_out[, c("age", "bmi")], verbosity = 0L)
conformal_metrics(region, held_out[["y"]])