scores_sample_univ: Scoring Rules for Simulated Forecast Distributions

Description

Calculate scores (CRPS, LogS, DSS) given observations and draws from the predictive distributions.

Usage

crps_sample(
  y,
  dat,
  method = "edf",
  w = NULL,
  bw = NULL,
  num_int = FALSE,
  show_messages = TRUE
)
logs_sample(y, dat, bw = NULL, show_messages = FALSE)
dss_sample(y, dat, w = NULL)

Value

Value of the score. A lower score indicates a better forecast.

Arguments

y: vector of realized values.
dat: vector or matrix (depending on y; see details) of simulation draws from forecast distribution.
method: string; approximation method. Options: "edf" (empirical distribution function) and "kde" (kernel density estimation).
w: optional; vector or matrix (matching dat) of weights for method "edf".
bw: optional; vector (matching y) of bandwidths for kernel density estimation; see details.
num_int: logical; if TRUE numerical integration is used for method "kde".
show_messages: logical; display of messages (does not affect warnings and errors).

Author

Alexander Jordan, Fabian Krueger, Sebastian Lerch

Details

For a vector y of length n >= 2, dat should be given as a matrix with n rows. If y has length 1, then dat may be a vector.

crps_sample employs an empirical version of the quantile decomposition of the CRPS (Laio and Tamea, 2007) when using method = "edf". For method = "kde", it uses kernel density estimation using a Gaussian kernel. The logarithmic score always uses kernel density estimation.

The bandwidth (bw) for kernel density estimation can be specified manually, in which case it must be a vector (matching y) of positive numbers. If bw == NULL, the bandwidth is selected using the core function bw.nrd. Numerical integration may speed up computation for crps_sample in case of large samples dat.

References

Evaluating simulation based forecast distributions:

Krueger, F., Lerch, S., Thorarinsdottir, T.L. and T. Gneiting (2021): `Predictive inference based on Markov chain Monte Carlo output', International Statistical Review 89, 274-301. tools:::Rd_expr_doi("10.1111/insr.12405")

Empirical quantile decomposition of the CRPS:

Laio, F. and S. Tamea (2007): `Verification tools for probabilistic forecasts of continuous hydrological variables', Hydrology and Earth System Sciences, 11, 1267-1277. tools:::Rd_expr_doi("10.5194/hess-11-1267-2007")

Examples

Run this code

if (FALSE) {

# y has length greater than 1
y <- 1:2
sample <- matrix(rnorm(20), nrow = 2)
crps_sample(y = y, dat = sample)
logs_sample(y = y, dat = sample)

y <- 1:2
sample <- rnorm(10)
crps_sample(y = y, dat = sample) # error

# y has length 1
y <- 1
sample <- rnorm(10)
crps_sample(y = y, dat = sample)

sample <- matrix(rnorm(10), nrow = 1)
crps_sample(y = y, dat = sample)

sample <- matrix(rnorm(20), nrow = 2)
crps_sample(y = y, dat = sample) # error
}

Run the code above in your browser using DataLab

Last chance! 50% off unlimited learning