verify: Verification function

Description

Based on the type of inputs, this function calculates a range of verification statistics and skill scores. Additionally, it creates a verify class object that can be used in further analysis or with other methods such as plot and summary.

Usage

verify(obs, pred, p = NULL, baseline = NULL,    frcst.type = "prob", obs.type = "binary",
    thresholds = seq(0,1,0.1), show = TRUE, bins = TRUE,
    fudge = 0.01, ...)

Arguments

obs

The values with which the verifications are verified. May be a vector of length 4 if the forecast and predictions are binary data summarized in a contingency table. In this case, the value are entered in the order of c(n11, n0

pred

Prediction of event. The prediction may be in the form of the a point prediction or the probability of a forecast. Let pred = NULL if obs is a contingency table.

the probability level of the quantile forecast, any value between 0 and 1.

baseline

In meteorology, climatology is the baseline that represents the no-skill forecast. In other fields this field would differ. This field is used to calculate certain skill scores. If left NULL, these statistics are calculated

frcst.type

Forecast type. One of "prob", "binary", "norm.dist", "cat" or "cont", or "quantile". Defaults to "prob". "norm.dist" is used when the forecast is in the form of a normal distribution. See crps for more details.

obs.type

Observation type. Either "binary", "cat" or "cont". Defaults to "binary"

thresholds

Thresholds to be considered for point forecasts of continuous events.

show

Binary; if TRUE (the default), print warning message

bins

Binary; if TRUE (default), the probabilistic forecasts are placed in bins defined by the sequence defined in threshold and assigned the midpoint value.

fudge

A numeric fudge factor to be added to each cell of the contingency table in order to avoid division by zero.

...

Additional options.

Value

An object of the verify class. Depending on the type of data used, the following information may be returned. The following notation is used to describe which values are produced for which type of forecast/observations. (BB = binary/binary, PB = probablistic/binary, CC = continuous/continuous, CTCT = categorical/categorical)
BSBrier Score (PB)
BSSBrier Skill Score(PB)
SSSkill Score (BB)
hit.rateHit rate, aka PODy, $h$ (PB, CTCT)
false.alarm.rateFalse alarm rate, PODn, $f$ (PB, CTCT)
TSThreat Score or Critical Success Index (CSI)(BB, CTCT)
ETSEquitable Threat Score (BB, CTCT)
BIASBias (BB, CTCT)
PCPercent correct or hit rate (BB, CTCT)
Cont.TableContingency Table (BB)
HSSHeidke Skill Score(BB, CTCT)
KSSKuniper Skill Score (BB)
PSSPierce Skill Score (CTCT)
GSGerrity Score (CTCT)
MEMean error (CC)
MSEMean-squared error (CC)
MAEMean absolute error (CC)
thetaOdds Ratio (BB)
log.thetaLog Odds Ratio
n.hDegrees of freedom for log.theta (BB)
orssOdds ratio skill score, aka Yules's Q (BB)
edsExtreme Dependency Score (BB)
eds.seStandard Error for Extreme Dependence Score (BB)
sedsSymmetric Extreme Dependency Score (BB)
seds.seStandard Error for Symmetric Extreme Dependency Score (BB)
EDIExtremal Dependence Index (BB)
EDI.seStandard Error for Extremal Dependence Index (BB)
SEDISymmetric Extremal Dependence Index (BB)
SEDI.seStandard Error for Symmetric Extremal Dependence Index (BB)

Details

See Wilks (2006) and the WMO Joint WWRP/WGNE Working Group web site on verification for more details about these verification statistics. See Stephenson et al. (2008) and Ferro and Stephenson (2011) for more on the extreme dependence scores and indices. For information on confidence intervals for these scores, see Gilleland (2010).

References

Ferro, C. A. T. and D. B. Stephenson, 2011. Extremal dependence indices: Improved verification measures for deterministic forecasts of rare binary events. Wea. Forecasting, 26, 699 - 713.

Gilleland, E., 2010. Confidence intervals for forecast verification. NCAR Technical Note NCAR/TN-479+STR, 71pp. Available at: http://nldr.library.ucar.edu/collections/technotes/asset-000-000-000-846.pdf Stephenson, D. B., B. Casati, C. A. T. Ferro, and C. A. Wilson, 2008. The extreme dependency score: A non-vanishing measure for forecasts of rare events. Meteor. Appl., 15, 41 - 50. Wilks, D. S., 2006. Statistical Methods in the Atmospheric Sciences, San Diego: Academic Press., 627 pp. (2nd Editiion).

WMO Joint WWRP/WGNE Working Group on Verification Website http://www.cawcr.gov.au/projects/verification/

Examples

Run this code

# binary/binary example
obs<- round(runif(100))
pred<- round(runif(100))

# binary/binary example
# Finley tornado data.

obs<- c(28, 72, 23, 2680)
A<- verify(obs, pred = NULL, frcst.type = "binary", obs.type = "binary")

summary(A)

# categorical/categorical example
# creates a simulated 5 category forecast and observation.
obs <- round(runif(100, 1,5) )
pred <- round(runif(100, 1,5) )

A<- verify(obs, pred, frcst.type = "cat", obs.type = "cat" )
summary(A)

#  probabilistic/ binary example

pred<- runif(100)
A<- verify(obs, pred, frcst.type = "prob", obs.type = "binary")
summary(A)

# continuous/ continuous example
obs<- rnorm(100)
pred<- rnorm(100)
baseline <- rnorm(100, sd = 0.5) 

A<- verify(obs, pred, baseline = baseline,  frcst.type = "cont", obs.type = "cont")
summary(A)

Run the code above in your browser using DataLab