This function calculates bin-conditional conformal prediction intervals with a confidence level of 1-alpha for a vector of (continuous) predicted values using inductive conformal prediction on a bin-by-bin basis. The intervals are computed using a calibration set with predicted and true values and their associated bins. The function returns a tibble containing the predicted values along with the lower and upper bounds of the prediction intervals. Bin-conditional conformal prediction intervals are useful when the prediction error is not constant across the range of predicted values and ensures that the coverage is (approximately) correct for each bin under the assumption that the non-conformity scores are exchangeable within each bin.
pinterval_bccp(
pred,
calib = NULL,
calib_truth = NULL,
calib_bins = NULL,
breaks = NULL,
right = TRUE,
contiguize = FALSE,
alpha = 0.1,
ncs_type = c("absolute_error", "relative_error", "za_relative_error",
"heterogeneous_error", "raw_error"),
grid_size = 10000,
resolution = NULL,
distance_weighted_cp = FALSE,
distance_features_calib = NULL,
distance_features_pred = NULL,
normalize_distance = TRUE,
distance_type = c("mahalanobis", "euclidean"),
weight_function = c("gaussian_kernel", "caucy_kernel", "logistic", "reciprocal_linear")
)A tibble with the predicted values and the lower and upper bounds of the prediction intervals. If contiguize = FALSE, the intervals may consist of multiple disjoint segments; in this case, the tibble will contain a list-column with all segments for each prediction.
Vector of predicted values
A numeric vector of predicted values in the calibration partition, or a 2 column tibble or matrix with the first column being the predicted values and the second column being the truth values. If calib is a numeric vector, calib_truth must be provided.
A numeric vector of true values in the calibration partition. Only required if calib is a numeric vector
A vector of bin identifiers for the calibration set. Not used if breaks are provided.
A vector of break points for the bins to manually define the bins. If NULL, lower and upper bounds of the bins are calculated as the minimum and maximum values of each bin in the calibration set. Must be provided if calib_bins is not provided, either as a vector or as the last column of a calib tibble.
Logical, if TRUE the bins are right-closed (a,b] and if FALSE the bins are left-closed `[ a,b)`. Only used if breaks are provided.
Logical indicating whether to contiguize the intervals. TRUE will consider all bins for each prediction using the lower and upper endpoints as interval limits to avoid non-contiguous intervals. FALSE will allows for non-contiguous intervals. TRUE guarantees at least appropriate coverage in each bin, but may suffer from over-coverage in certain bins. FALSE will have appropriate coverage in each bin but may have non-contiguous intervals. Default is FALSE.
The confidence level for the prediction intervals. Must be a single numeric value between 0 and 1
A string specifying the type of nonconformity score to use. Available options are:
"absolute_error": \(|y - \hat{y}|\)
"relative_error": \(|y - \hat{y}| / \hat{y}\)
"zero_adjusted_relative_error": \(|y - \hat{y}| / (\hat{y} + 1)\)
"heterogeneous_error": \(|y - \hat{y}| / \sigma_{\hat{y}}\) absolute error divided by a measure of heteroskedasticity, computed as the predicted value from a linear model of the absolute error on the predicted values
"raw_error": the signed error \(y - \hat{y}\)
The default is "absolute_error".
The number of points to use in the grid search between the lower and upper bound. Default is 10,000. A larger grid size increases the resolution of the prediction intervals but also increases computation time.
Alternatively to grid_size. The minimum step size between grid points. Useful if the a specific resolution is desired. Default is NULL.
Logical. If TRUE, weighted conformal prediction is performed where the non-conformity scores are weighted based on the distance between calibration and prediction points in feature space. Default is FALSE. See details for more information.
A matrix, data frame, or numeric vector of features from which to compute distances when distance_weighted_cp = TRUE. This should contain the feature values for the calibration set. Must have the same number of rows as the calibration set. Can be the predicted values themselves, or any other features which give a meaningful distance measure.
A matrix, data frame, or numeric vector of feature values for the prediction set. Must be the same features as specified in distance_features_calib. Required if distance_weighted_cp = TRUE.
Either 'minmax', 'sd', or 'none'. Indicates if and how to normalize the distances when distance_weighted_cp is TRUE. Normalization helps ensure that distances are on a comparable scale across features. Default is 'none'.
The type of distance metric to use when computing distances between calibration and prediction points. Options are 'mahalanobis' (default) and 'euclidean'.
A character string specifying the weighting kernel to use for distance-weighted conformal prediction. Options are:
"gaussian_kernel": \( w(d) = e^{-d^2} \)
"caucy_kernel": \( w(d) = 1/(1 + d^2) \)
"logistic": \( w(d) = 1//(1 + e^{d}) \)
"reciprocal_linear": \( w(d) = 1/(1 + d) \)
The default is "gaussian_kernel". Distances are computed as the Euclidean distance between the calibration and prediction feature vectors.
`pinterval_bccp()` extends [pinterval_conformal()] to the bin-conditional setting, where prediction intervals are calibrated separately within user-specified bins. It is particularly useful when prediction error varies across the range of predicted values, as it enables locally valid coverage by ensuring that the coverage level \(1 - \alpha\) holds within each bin—assuming exchangeability of non-conformity scores within bins.
For a detailed description of non-conformity scores, distance weighting and the general inductive conformal framework, see [pinterval_conformal()].
For `pinterval_bccp()`, the calibration set must include predicted values, true values, and corresponding bin identifiers or breaks for the bins. These can be provided either as separate vectors (`calib`, `calib_truth`, and `calib_bins` or `breaks`).
Bins endpoints can be defined manually via the `breaks` argument or inferred from the calibration data. If `contiguize = TRUE`, the function ensures the resulting prediction intervals are contiguous across bins, potentially increasing coverage beyond the nominal level in some bins. If `contiguize = FALSE`, the function may produce non-contiguous intervals, which are more efficient but may be harder to interpret.
pinterval_conformal
# Generate example data
library(dplyr)
library(tibble)
x1 <- runif(1000)
x2 <- runif(1000)
y <- rlnorm(1000, meanlog = x1 + x2, sdlog = 0.5)
# Create bins based on quantiles
bin <- cut(y, breaks = quantile(y, probs = seq(0, 1, 1/4)),
include.lowest = TRUE, labels =FALSE)
df <- tibble(x1, x2, y, bin)
df_train <- df %>% slice(1:500)
df_cal <- df %>% slice(501:750)
df_test <- df %>% slice(751:1000)
# Fit a model to the training data
mod <- lm(log(y) ~ x1 + x2, data=df_train)
# Generate predictions on the original y scale for the calibration data
calib <- exp(predict(mod, newdata=df_cal))
calib_truth <- df_cal$y
calib_bins <- df_cal$bin
# Generate predictions for the test data
pred_test <- exp(predict(mod, newdata=df_test))
# Calculate bin-conditional conformal prediction intervals
pinterval_bccp(pred = pred_test,
calib = calib,
calib_truth = calib_truth,
calib_bins = calib_bins,
alpha = 0.1)
Run the code above in your browser using DataLab