scoringutils (version 2.3.0)

score.forecast_binary: Evaluate forecasts

Description

score() applies a selection of scoring metrics to a forecast object. score() is a generic that dispatches to different methods depending on the class of the input data.

See as_forecast_binary(), as_forecast_quantile() etc. for information on how to create a forecast object.

See get_forecast_unit() for more information on the concept of a forecast unit.

For additional help and examples, check out the paper Evaluating Forecasts with scoringutils in R.

Usage

# S3 method for forecast_binary
score(forecast, metrics = get_metrics(forecast), ...)
# S3 method for forecast_multivariate_point
score(forecast, metrics = get_metrics(forecast), ...)
# S3 method for forecast_multivariate_sample
score(forecast, metrics = get_metrics(forecast), ...)
# S3 method for forecast_nominal
score(forecast, metrics = get_metrics(forecast), ...)
# S3 method for forecast_ordinal
score(forecast, metrics = get_metrics(forecast), ...)
# S3 method for forecast_point
score(forecast, metrics = get_metrics(forecast), ...)
# S3 method for forecast_quantile
score(forecast, metrics = get_metrics(forecast), ...)
# S3 method for forecast_sample
score(forecast, metrics = get_metrics(forecast), ...)
score(forecast, metrics, ...)

Value

An object of class scores. This object is a data.table with unsummarised scores (one score per forecast) and has an additional attribute metrics with the names of the metrics used for scoring. See summarise_scores()) for information on how to summarise scores.

Arguments

forecast

A forecast object (a validated data.table with predicted and observed values).

metrics

A named list of scoring functions. Each element should be a function reference, not a function call. For example, use list("crps" = crps_sample) rather than list("crps" = crps_sample()). Names will be used as column names in the output. See get_metrics() for more information on the default metrics used. See the Customising metrics section below for information on how to pass custom arguments to scoring functions.

...

Currently unused. You cannot pass additional arguments to scoring functions via .... See the Customising metrics section below for details on how to use purrr::partial() to pass arguments to individual metrics.

Author

Nikos Bosse [email protected]

Details

Customising metrics

If you want to pass arguments to a scoring function, you need change the scoring function itself via e.g. purrr::partial() and pass an updated list of functions with your custom metric to the metrics argument in score(). For example, to use interval_coverage() with interval_range = 90, you would define a new function, e.g. interval_coverage_90 <- purrr::partial(interval_coverage, interval_range = 90) and pass this new function to metrics in score().

Note that if you want to pass a variable as an argument, you can unquote it with !! to make sure the value is evaluated only once when the function is created. Consider the following example:

custom_arg <- "foo"
print1 <- purrr::partial(print, x = custom_arg)
print2 <- purrr::partial(print, x = !!custom_arg)
custom_arg <- "bar"
print1() # prints 'bar'
print2() # prints 'foo'

References

Bosse NI, Gruson H, Cori A, van Leeuwen E, Funk S, Abbott S (2022) Evaluating Forecasts with scoringutils in R. tools:::Rd_expr_doi("10.48550/arXiv.2205.07090")

Examples

# \dontshow{
  data.table::setDTthreads(2) # restricts number of cores used on CRAN
# }

validated <- as_forecast_quantile(example_quantile)
score(validated) |>
  summarise_scores(by = c("model", "target_type"))

# set forecast unit manually (to avoid issues with scoringutils trying to
# determine the forecast unit automatically)
example_quantile |>
  as_forecast_quantile(
    forecast_unit = c(
      "location", "target_end_date", "target_type", "horizon", "model"
    )
  ) |>
  score()

# forecast formats with different metrics
if (FALSE) {
score(as_forecast_binary(example_binary))
score(as_forecast_quantile(example_quantile))
score(as_forecast_point(example_point))
score(as_forecast_sample(example_sample_discrete))
score(as_forecast_sample(example_sample_continuous))
}

# passing a subset of metrics using select_metrics()
# (the preferred approach for selecting from default metrics)
example_sample_continuous |>
  as_forecast_sample() |>
  score(metrics = select_metrics(
    get_metrics(as_forecast_sample(example_sample_continuous)),
    select = c("crps", "mad")
  ))

# passing a custom list of metrics manually
# make sure to pass the function itself, not the result of calling it,
# i.e. use `crps_sample` (correct) instead of `crps_sample()` (incorrect)
example_sample_continuous |>
  as_forecast_sample() |>
  score(metrics = list("crps" = crps_sample, "mad" = mad_sample))

# multivariate forecasts
if (FALSE) {
score(example_multivariate_sample)
}