This function performs the analyses described in Gilleland and Roux (2014). Namely, the Diebold Mariano test for competing forecast performance, the Hering and Genton (2011) modification of this test, as well as the dynamic time warping extension.The Diebold-Mariano test was proposed in Diebold and Mariano (1995) for obtaining hypothesis tests to compare the forecast accuracy of two competing forecasts against a common verification series. The null hypothesis is that they have the same accuracy. The test statistic is of the form
S = dbar/sqrt(2*pi*se_{d}(0)/N),
where d is the loss differential, d = e1 - e2 (e1 = loss(x, xhat1) and e2 = loss(x, xhat2)), dbar is its sample mean, and se_{d}(0) is the standard error for d, which must be estimated, and N is the length of the series investigated. Let V = 2*pi*se_{d}(0), then V is estimated by
V = sum(gamma(tau)),
where the summation is over tau = -(k - 1) to (k - 1) for temporal lags k, and gamma are the empirical autocovariances.
Hering and Genton (2011) propose a modification that employs fitting a parameteric covariance model in determining the standard error for the test statistic (they also propose a spatial extension, see, e.g., spatMLD from SpatialVx). If fitmodel is not NULL, then this is the method applied, and the resulting standard error is calculating by summing up the values of this function up to half the maximum lag.
In either case, asymptotic results suggest that S ~ N(0,1), and the hypothesis test is conducted subsequently.
Discrete time warping can be applied (see examples below) in order to obtain a loss function based on location (in time) and intensity errors similar to the spatial version in Gilleland (2013).
The loss functions supplied by losserr include:
abserr: Absolute error loss, defined by abs((xhat - x)/scale),
sqerr: Square error loss, defined by ((xhat - x)/scale)^2,
simple: Simple loss, defined by (xhat - x)/scale,
power: Power loss, defined by ((xhat - x)/scale)^p (same as sqerr if p=2),
corrskill: Correlation skill defined by scale * (x - mean(x)) * (xhat - mean(xhat)),
dtw: Discrete time warp loss defined by: d1 + d2, where d1 is the absolute distance (ignoring direction) of warp movement, and d2 is one of the above loss functions (except for corrskill) applied to the resulting intensity errors after warping the series.
The exponentialACV function takes numeric vector arguments x and y and estimates the parameters, c(sigma, theta), that optimize
y = sigma^2*exp(-3*x/theta)
using nls. If fitmodel = exponentialACV, then this exponential function is fit to the data set where x are lags and y are empirical autocovariances (found by a call to acf).