forecTheta (version 3.0.3)

PI_eval: Prediction Interval (PI) evaluation

Description

Implementation of Mean Scaled Interval Score (MSIS) and Absolute Coverage Difference (ACD).

Usage

PI_eval(obs, forec, lower_bounds, upper_bounds, name = c("MSIS", "ACD"), alpha = 0.05)

Value

Returns a numeric value corresponding to the selected evaluation metric (MSIS or ACD).

Arguments

obs

Observed time series values.

forec

Forecasted values corresponding to the observed data.

lower_bounds

Lower bounds of the prediction interval.

upper_bounds

Upper bounds of the prediction interval.

name

Evaluation metric to be used. Options are "MSIS" or "ACD".

alpha

Significance level for the prediction interval. Default is 0.05.

Author

Jose Augusto Fiorucci, Francisco Louzada, Igor De Oliveira Barros Faluhelyi.

Details

The MSIS is computed by combining the width of the prediction interval with a penalty for observations falling outside the interval. This score is averaged over time and scaled by the mean absolute seasonal difference, following the same approach used for the MASE in the M4 competition, to ensure scale invariance. As a complementary metric, the ACD measures the absolute difference between the empirical and nominal coverage (e.g., 0.95), helping to assess interval reliability.

References

Gneiting, T. and Raftery, A.E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102 (477), 359--378, tools:::Rd_expr_doi("10.1198/016214506000001437").

Makridakis, S., Spiliotis, E. and Assimakopoulos, V. (2018). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36 (1), 54--74, tools:::Rd_expr_doi("10.1016/j.ijforecast.2019.04.014").

See Also

forecTheta-package, errorMetricFunctions, dotm, dstm, otm, stm

Examples

y1 <- 2 + 0.15 * (1:20) + rnorm(20)
y2 <- y1[20] + 0.3 * (1:30) + rnorm(30)
y <- as.ts(c(y1, y2))
out <- dotm(y, h = 10)

# MSIS metric
PI_eval(
  obs = y, forec = out$mean,
  lower_bounds = out$lower[, 3], upper_bounds = out$upper[, 3],
  name = "MSIS", alpha = 0.05
)

# ACD metric
PI_eval(
  obs = y, forec = out$mean,
  lower_bounds = out$lower[, 3], upper_bounds = out$upper[, 3],
  name = "ACD", alpha = 0.05
)