caret (version 6.0-57)

calibration: Probability Calibration Plot

Description

For classification models, this function creates a 'calibration plot' that describes how consistent model probabilities are with observed event rates.

Usage

calibration(x, ...)

## S3 method for class 'formula': calibration(x, data = NULL, class = NULL, cuts = 11, subset = TRUE, lattice.options = NULL, ...)

## S3 method for class 'calibration': xyplot(x, data, ...)

panel.calibration(...)

Arguments

x
a lattice formula (see xyplot for syntax) where the left-hand side of the formula is a factor class variable of the observed outcome and the right-hand side specifies one or model
data
For calibration.formula, a data frame (or more precisely, anything that is a valid envir argument in eval, e.g., a list or an environment) containing values for any variables in the formula, as well as groups
class
a character string for the class of interest
cuts
If a single number this indicates the number of splits of the data are used to create the plot. By default, it uses as many cuts as there are rows in data. If a vector, these are the actual cuts that will be used.
subset
An expression that evaluates to a logical or integer indexing vector. It is evaluated in data. Only the resulting rows of data are used for the plot.
lattice.options
A list that could be supplied to lattice.options
...
options to pass through to xyplot or the panel function (not used in calibration.formula).

Value

  • calibration.formula returns a list with elements:
  • datathe data used for plotting
  • cutsthe number of cuts
  • classthe event class
  • probNamesthe names of the model probabilities
  • xyplot.calibration returns a lattice object

Details

calibration.formula is used to process the data and xyplot.calibration is used to create the plot.

To construct the calibration plot, the following steps are used for each model:

  1. The data are split intocuts - 1roughly equal groups by their class probabilities
  2. the number of samples with true results equal toclassare determined
  3. the event rate is determined for each bin
xyplot.calibration produces a plot of the observed event rate by the mid-point of the bins.

This implementation uses the lattice function xyplot, so plot elements can be changed via panel functions, trellis.par.set or other means. calibration uses the panel function panel.calibration by default, but it can be changed by passing that argument into xyplot.calibration.

The following elements are set by default in the plot but can be changed by passing new values into xyplot.calibration: xlab = "Bin Midpoint", ylab = "Observed Event Percentage", type = "o", ylim = extendrange(c(0, 100)),xlim = extendrange(c(0, 100)) and panel = panel.calibration

See Also

xyplot, trellis.par.set

Examples

Run this code
data(mdrr)
mdrrDescr <- mdrrDescr[, -nearZeroVar(mdrrDescr)]
mdrrDescr <- mdrrDescr[, -findCorrelation(cor(mdrrDescr), .5)]


inTrain <- createDataPartition(mdrrClass)
trainX <- mdrrDescr[inTrain[[1]], ]
trainY <- mdrrClass[inTrain[[1]]]
testX <- mdrrDescr[-inTrain[[1]], ]
testY <- mdrrClass[-inTrain[[1]]]

library(MASS)

ldaFit <- lda(trainX, trainY)
qdaFit <- qda(trainX, trainY)

testProbs <- data.frame(obs = testY,
                        lda = predict(ldaFit, testX)$posterior[,1],
                        qda = predict(qdaFit, testX)$posterior[,1])

calibration(obs ~ lda + qda, data = testProbs)

calPlotData <- calibration(obs ~ lda + qda, data = testProbs)
calPlotData

xyplot(calPlotData, auto.key = list(columns = 2))

Run the code above in your browser using DataCamp Workspace