scoringutils (version 0.1.7.2)

eval_forecasts_sample: Evaluate forecasts in a Sample-Based Format (Integer or Continuous)

Description

Evaluate forecasts in a Sample-Based Format (Integer or Continuous)

Usage

eval_forecasts_sample(
  data,
  by,
  summarise_by,
  metrics,
  prediction_type,
  quantiles,
  sd,
  pit_plots,
  summarised,
  verbose
)

Arguments

data

A data.frame or data.table with the predictions and observations. Note: it is easiest to have a look at the example files provided in the package and in the examples below. The following columns need to be present:

• true_value - the true observed values

• prediction - predictions or predictive samples for one true value. (You only don't need to provide a prediction column if you want to score quantile forecasts in a wide range format.)

For integer and continuous forecasts a sample column is needed:

• sample - an index to identify the predictive samples in the prediction column generated by one model for one true value. Only necessary for continuous and integer forecasts, not for binary predictions.

For quantile forecasts the data can be provided in variety of formats. You can either use a range-based format or a quantile-based format. (You can convert between formats using quantile_to_range_long, range_long_to_quantile, sample_to_range_long, sample_to_quantile) For a quantile-format forecast you should provide:

• prediction - prediction to the corresponding quantile

• quantile - quantile to which the prediction corresponds

For a range format (long) forecast you need

• prediction the quantile forecasts

• boundary values should be either "lower" or "upper", depending on whether the prediction is for the lower or upper bound of a given range

• range the range for which a forecast was made. For a 50% interval the range should be 50. The forecast for the 25% quantile should have the value in the prediction column, the value of range should be 50 and the value of boundary should be "lower". If you want to score the median (i.e. range = 0), you still need to include a lower and an upper estimate, so the median has to appear twice.

Alternatively you can also provide the format in a wide range format. This format needs

• pairs of columns called something like 'upper_90' and 'lower_90', or 'upper_50' and 'lower_50', where the number denotes the interval range. For the median, you need to provide columns called 'upper_0' and 'lower_0'

by

character vector of columns to group scoring by. This should be the lowest level of grouping possible, i.e. the unit of the individual observation. This is important as many functions work on individual observations. If you want a different level of aggregation, you should use summarise_by to aggregate the individual scores. Also not that the pit will be computed using summarise_by instead of by

summarise_by

character vector of columns to group the summary by. By default, this is equal to `by` and no summary takes place. But sometimes you may want to to summarise over categories different from the scoring. summarise_by is also the grouping level used to compute (and possibly plot) the probability integral transform(pit).

metrics

the metrics you want to have in the output. If `NULL` (the default), all available metrics will be computed.

prediction_type

character, should be either "continuous" or "integer"

quantiles

numeric vector of quantiles to be returned when summarising. Instead of just returning a mean, quantiles will be returned for the groups specified through `summarise_by`. By default, no quantiles are returned.

sd

if TRUE (the default is FALSE) the standard deviation of all metrics will be returned when summarising.

pit_plots

if TRUE (not the default), pit plots will be returned. For details see pit.

summarised

Summarise arguments (i.e. take the mean per group specified in group_by. Default is TRUE.

verbose

print out additional helpful messages (default is TRUE)

Value

A data.table with appropriate scores. For more information see eval_forecasts

References

Funk S, Camacho A, Kucharski AJ, Lowe R, Eggo RM, Edmunds WJ (2019) Assessing the performance of real-time epidemic forecasts: A case study of Ebola in the Western Area region of Sierra Leone, 2014-15. PLoS Comput Biol 15(2): e1006785. <doi:10.1371/journal.pcbi.1006785>

Examples

# NOT RUN {
## Integer Forecasts
integer_example <- data.table::setDT(scoringutils::integer_example_data)
eval <- scoringutils::eval_forecasts(integer_example,
                                     summarise_by = c("model"),
                                     quantiles = c(0.1, 0.9),
                                     sd = TRUE,
                                     pit_plots = TRUE)
eval <- scoringutils::eval_forecasts(integer_example)

## Continuous Forecasts
continuous_example <- data.table::setDT(scoringutils::continuous_example_data)
eval <- scoringutils::eval_forecasts(continuous_example)#'

eval <- scoringutils::eval_forecasts(continuous_example,
                                     quantiles = c(0.5, 0.9),
                                     sd = TRUE,
                                     summarise_by = c("model"))

# }