rtemis (version 1.4.3)

conformal: Conformal prediction regions

Description

A prediction interval for a regression or a set of labels for a classification, covering the truth with probability at least 1 - alpha under exchangeability alone -- in finite samples, for any model, including a misspecified one.

Usage

conformal(x, newdata, calibration = NULL, config = NULL, verbosity = 1L, ...)

Value

PredictionInterval object for a regression, PredictionSet for a classification.

Arguments

x

Supervised or SupervisedRes object.

newdata

tabular data: Cases to bound. Predictors only, in training order, as stats::predict() requires.

calibration

Optional tabular data: Calibration cases, predictors and outcome, in the shape train() was given. NULL uses the model's stored test split where it has one.

config

Optional ConformalConfig object: Built by setup_SplitConformal, setup_CVPlus or setup_CQR. NULL takes the method the object supports.

verbosity

Integer: Verbosity level.

...

Additional arguments passed to methods.

Author

EDG

Details

The guarantee is marginal. P(Y in C(X)) >= 1 - alpha averages over the draw of both the calibration set and the test case. It is not conditional: 90% marginal coverage is compatible with 99% coverage on easy cases and 60% on a hard subgroup. The guarantee rests on exchangeability and does not hold under distribution shift.

Width is the quality measure; coverage is the correctness check. A model with no predictive value attains valid coverage with intervals wide enough to be uninformative, so the two are read together. The region reports its widths (or set sizes); conformal_metrics() scores coverage against outcomes the region did not see.

Which data calibrates. calibration is a tabular dataset holding the predictors and the outcome, in training shape. Left NULL, a model trained with dat_test calibrates on that split, whose residuals it already stores: train() never fits, tunes or early-stops on dat_test. Nothing else is assumed clean -- a validation split may have driven early stopping, and a model chosen by its test metric has used that split for selection, which nothing in the object records.

Conformal calibration is unrelated to the probability calibration calibrate performs, and the two must not share rows. rtemis raises an error where it detects shared rows.

Which method. config NULL takes the method the object supports: split conformal for a Supervised, CV+ for a SupervisedRes. See setup_SplitConformal, setup_CVPlus and setup_CQR.

Examples

x <- data.frame(age = rnorm(300), bmi = rnorm(300))
x[["y"]] <- x[["age"]] * 2 + rnorm(300, sd = 0.3)
# Three splits: one fits, one calibrates, one scores what the other two did.
mod <- train(
  x[1:200, ],
  dat_test = x[201:250, ],
  hyperparameters = setup_GLM(),
  verbosity = 0L
)

# The stored test split calibrates; 90% intervals for five fresh cases.
region <- conformal(mod, x[1:5, c("age", "bmi")], verbosity = 0L)
region
region@lower
region@upper

# Coverage, against outcomes nothing in the pipeline has seen.
held_out <- x[251:300, ]
conformal_metrics(
  conformal(mod, held_out[, c("age", "bmi")], verbosity = 0L),
  held_out[["y"]]
)