scoringRules (version 0.9.2)

sample: Scoring Rules for Simulated Forecast Distributions

Description

Compute scores of the form \(S(y, dat)\), where \(S\) is a proper scoring rule, \(y\) is a scalar realization and \(dat\) is a simulated sample of data.

Usage

crps_sample(y, dat, method = "edf", w = NULL, bw = NULL, 
            num_int = FALSE, show_messages = TRUE)
logs_sample(y, dat, bw = NULL, show_messages = TRUE)

Arguments

y

Realized value (numeric vector of length one).

dat

Numeric vector of data (simulation draws from forecast distribution).

method

String, approximation method used in crps_sample. Currently implemented: "edf" (empirical distribution function) and "kde" (kernel density estimation).

w

Numeric vector of observation weights for dat.

bw

Bandwidth parameter for kernel density estimator (numeric vector of length one).

num_int

Logical of length one, whether or not to use numerical integration (may speed up computation for crps_sample in case of large samples dat if method = "kde").

show_messages

Logical of length one, on/off switch for messages (does not affect warnings and errors, which are always printed).

Value

Value of the score. A lower score indicates a better forecast.

Details

When using method = "edf", crps_sample employs an empirical version of the quantile decomposition of the CRPS (Laio and Tamea, 2007). When using method = "kde", the function does kernel density estimation using a Gaussian kernel. The bandwidth (bw) can be specified manually, in which case it must be a positive number. If bandwidth == NULL, the bandwidth is selected using the core function bw.nrd.

logs_sample always uses kernel density estimation with a Gaussian kernel (the EDF option is not available since the score requires a density). The choice of bandwidth is handled as in crps_sample.

References

Evaluating simulation based forecast distributions:

Krueger, F., S. Lerch, T.L. Thorarinsdottir and T. Gneiting (2016), `Probabilistic forecasting and comparative model assessment based on Markov Chain Monte Carlo output', working paper, Heidelberg Institute for Theoretical Studies, available at http://arxiv.org/abs/1608.06802.

Empirical quantile decomposition of the CRPS:

Laio, F. and S. Tamea (2007), `Verification tools for probabilistic forecasts of continuous hydrological variables', Hydrology and Earth System Sciences, 11, 1267-1277.

Examples

# NOT RUN {
sample <- rnorm(1e4)
obs <- 1
crps_sample(y = obs, dat = sample)  
logs_sample(y = obs, dat = sample)

## exact solutions using analytical closed form expression:
crps(y = obs, family = "normal", mean = 0, sd = 1) 
logs(y = obs, family = "normal", mean = 0, sd = 1)
# }