Functions to calculate a number of performance metrics.
mape(actual, predicted)bias(actual, predicted)
mslre(actual, predicted)
mase(actual, predicted, original_series = NULL, frequency = 1)
mis(actual, lower, upper, alpha)
wape(actual, predicted, weights)
wslre(actual, predicted, weights)
wse(actual, predicted, weights)
pinball(actual, distribution, alpha = 0.1)
crps(actual, distribution)
rmape(actual, predicted)
smape(actual, predicted)
msis(actual, lower, upper, original_series, frequency = 1, alpha)
A numeric value.
the actual values corresponding to the forecast period.
the predicted values corresponding to the forecast period.
the actual values corresponding to the training period.
the seasonal frequency of the series used in the model.
the lower distributional forecast for the quantile corresponding to the coverage ratio alpha (i.e. alpha/2).
the upper distributional forecast for the quantile corresponding to the coverage ratio alpha (i.e. 1 - alpha/2).
the distributional coverage.
a vector of weights for generating weighted metrics. If the actual and predicted inputs are univariate, this should be equal to the length of the actual series and calculates a time-weighted average; otherwise, the weights should be of length equal to the number of series in a multivariate case, in which case a cross-sectional average is calculated.
the forecast distribution (returned in the distribution slot of the prediction object). This is used in the continuous ranked probability score (crps) of Gneiting et al. (2005), and calculated using the function from the `scoringRules` package.
Alexios Galanos
The following performance metrics are implemented:
Measures the average percentage deviation of predictions from actual values. $$ MAPE = \frac{1}{n} \sum_{t=1}^{n} \left| \frac{A_t - P_t}{A_t} \right| $$ where \(A_t\) is the actual value and \(P_t\) is the predicted value.
A transformation of MAPE using a Box-Cox transformation for scale invariance (Swanson et al.).
An alternative to MAPE that symmetrizes the denominator. $$ SMAPE = \frac{2}{n} \sum_{t=1}^{n} \frac{|A_t - P_t|}{|A_t| + |P_t|} $$
Compares the absolute error to the mean absolute error of a naive seasonal forecast. $$ MASE = \frac{\frac{1}{n} \sum_{t=1}^{n} |P_t - A_t|}{\frac{1}{N-s} \sum_{t=s+1}^{N} |A_t - A_{t-s}|} $$ where \(s\) is the seasonal period.
Measures squared log relative errors to penalize large deviations. $$ MSLRE = \frac{1}{n} \sum_{t=1}^{n} \left( \log(1 + A_t) - \log(1 + P_t) \right)^2 $$
Evaluates the accuracy of prediction intervals. $$ MIS = \frac{1}{n} \sum_{t=1}^{n} (U_t - L_t) + \frac{2}{\alpha} [(L_t - A_t) I(A_t < L_t) + (A_t - U_t) I(A_t > U_t)] $$ where \(L_t\) and \(U_t\) are the lower and upper bounds of the interval.
A scaled version of MIS, dividing by the mean absolute seasonal error. $$ MSIS = \frac{1}{h} \sum_{t=1}^{h} \frac{(U_t - L_t) + \frac{2}{\alpha} [(L_t - A_t) I(A_t < L_t) + (A_t - U_t) I(A_t > U_t)]}{\frac{1}{N-s} \sum_{t=s+1}^{N} |A_t - A_{t-s}|} $$
Measures systematic overestimation or underestimation. $$ Bias = \frac{1}{n} \sum_{t=1}^{n} (P_t - A_t) $$
A weighted version of MAPE. $$ WAPE = \sum_{t=1}^{n} \mathbf{w} \frac{|P_t - A_t|}{A_t} $$ where \(\mathbf{w}\) is the weight vector.
A weighted version of squared log relative errors. $$ WSLRE = \sum_{t=1}^{n} \mathbf{w} (\log(P_t / A_t))^2 $$
A weighted version of squared errors. $$ WSE = \sum_{t=1}^{n} \mathbf{w} \left( \frac{P_t}{A_t} \right)^2 $$
A scoring rule used for quantile forecasts. $$ \text{Pinball} = \frac{1}{n} \sum_{t=1}^{n} \left[ \tau (A_t - Q^\tau_t) I(A_t \geq Q^\tau_t) + (1 - \tau) (Q^\tau_t - A_t) I(A_t < Q^\tau_t) \right] $$ where $$Q^\tau_t$$ is the predicted quantile at level $$\tau$$.
A measure of probabilistic forecast accuracy. $$ CRPS = \frac{1}{n} \sum_{t=1}^{n} \int_{-\infty}^{\infty} (F_t(y) - I(y \geq A_t))^2 dy $$ where \(F_t(y)\) is the cumulative forecast distribution.
Tofallis2015tsaux
Hyndman2006tsaux
Gneiting2005tsaux
Gneiting2007tsaux
Swanson2011tsaux