fitdistcp (version 0.1.1)

reltest2: Evaluation of Reliability for Certain Additional Models in the fitdistcp Package

Description

This routine is mainly for reproducing certain results in Jewson et al. (2025), and not of general interest.

It uses simulations to evaluate the reliability of the predictive quantiles produced by the qgev_cp, ggpd_cp and qgev_p1_cp routines in the fitdistcp package. For each model, results for 5 models are calculated. This is to illustrate that the calibrating prior predictions dominate the ml, flat, crhp_ml and jp predictions, in terms of reliability.

Usage

reltest2(
  model = "gev",
  ntrials = 100,
  nrepeats = 3,
  nx = 50,
  params = c(0, 1, 0),
  alpha = seq(0.005, 0.995, 0.005),
  plotflag = TRUE,
  verbose = TRUE
)

Value

A plot showing 9 different reliability checks, and a list containing various outputs, including the probabilities shown in the plot.

Arguments

model

which distribution to test. Possibles values are "gev", "gpd_k1", "gev_p1".

ntrials

the number of trials to run. 5000 typically gives good results.

nrepeats

the number of entire repeats of the test to run, to check for convergence. 3 is a good choice.

nx

the length of the training data.

params

values for the parameters for the specified distribution

alpha

the alpha values at which to test

plotflag

logical to turn the plotting on and off

verbose

logical to turn loop counting on and off

Author

Stephen Jewson

Details

The maximum likelihood quantiles (plotted in blue) do not give good reliability. They typically underestimate the tails (see panel (f)).

The cp predictive quantiles generally give reasonably good reliability, especially for sample sizes of ~100. The other predictions generally give poor reliability.

References

If you use this package, we would be grateful if you would cite the following reference, which gives the various calibrating priors, and tests them for reliability:

• Jewson S., Sweeting T. and Jewson L. (2024): Reducing Reliability Bias in Assessments of Extreme Weather Risk using Calibrating Priors; ASCMO Advances in Statistical Climatology, Meteorology and Oceanography), https://ascmo.copernicus.org/articles/11/1/2025/.

See Also

An introduction to fitdistcp, with more examples, is given on this webpage.

The fitdistcp package currently includes the following models (in alphabetical order):

• Cauchy (cauchy),

• Cauchy with linear predictor on the mean (cauchy_p1),

• Exponential (exp),

• Exponential with log-linear predictor on the scale (exp_p1),

• Frechet with known location parameter (frechet_k1),

• Frechet with log-linear predictor on the scale and known location parameter (frechet_p2k1),

• Gamma (gamma),

• Generalized normal (gnorm),

• GEV (gev),

• GEV with linear predictor on the location (gev_p1),

• GEV with linear predictor on the location and log-linear prediction on the scale (gev_p12),

• GEV with linear predictor on the location, log-linear prediction on the scale, and linear predictor on the shape (gev_p123),

• GEV with linear predictor on the location and known shape (gev_p1k3),

• GEV with known shape (gev_k3),

• GPD with known location (gpd_k1),

• Gumbel (gumbel),

• Gumbel with linear predictor on the mean(gumbel_p1),

• Half-normal (halfnorm),

• Inverse gamma (invgamma),

• Inverse Gaussian (invgauss),

• t distribution with unknown location and scale and known DoF (lst_k3),

• t distribution with unknown location and scale, linear predictor on the location, and known DoF (lst_p1k3),

• Logistic (logis),

• Logistic with linear predictor on the location (logis_p1),

• Log-normal (lnorm),

• Log-normal with linear predictor on the location (lnorm_p1),

• Normal (norm),

• Normal with linear predictor on the mean (norm_p1),

• Pareto with known scale (pareto_k2),

• Pareto with log-linear predictor on the shape and known scale (pareto_p1k2),

• Uniform (unif),

• Weibull (weibull),

• Weibull with linear predictor on the scale (weibull_p2),

The level of predictive probability matching achieved by the maximum likelihood and calibrating prior quantiles, for any model, sample size and true parameter values, can be demonstrated using the routine reltest.

Model selection among models can be demonstrated using the routines ms_flat_1tail, ms_flat_2tail, ms_predictors_1tail, and ms_predictors_2tail,

Examples

set.seed(1)
# example 1
# -runs the default settings, which test reliability for the GEV distribution
reltest2(nrepeats=1)