CORElearn (version 0.9.29)

calibrate: Calibration of probabilities according to the given prior.

Description

Given probability scores predictedProb as provided for example by a call to predict.CoreModel and using one of available methods given by methods the function calibrates predicted probabilities so that they match the actual probabilities of a binary class 1 provided by correctClass.

Usage

calibrate(correctClass, predictedProb, class1=1, method = c("isoReg","binIsoReg","binning","chiMerge"), weight=NULL,noBins=10)

Arguments

correctClass

A vector of correct class labels for a binary classification problem.

predictedProb

A vector of predicted class 1 probability scores of the same length as correctClass.

class1

A class value (factor) or an index of the class value to be taken as a class to be calibrated.

method

One of isoReg, binIsoReg, binning, or chiMerge. See details below.

weight

If specified, should be of the same length as correctClass and gives weights for all the instances, otherwise a default weight of 1 for each instance is assumed.

noBins

If model="binning" or model="binIsoReg" specifies desired number of bins i.e., calibration bands.

Value

• A function returns a list with two vector components of the same length:
• interval
  The boundaries of the intervals. Lower boundary 0 is not explicitly included but should be taken into account.
• calProb
  The calibrated probabilities for each corresponding interval.

Details

Depending on the specified method one of the following calibration methods is executed.

• "isoReg"isotonic regression calibration based on pair-adjacent violators (PAV) algorithm.
• "binning"calibration into a pre-specified number of bands given bynoBinsparameter.
• "binIsoReg"first binning method is executed, following by a isotonic regression calibration.
• "chiMerge"first intervals are merged by a chi-squared statistics criterion, following by the isotonic regression calibration.

References

I. Kononenko, M. Kukar: Machine Learning and Data Mining: Introduction to Principles and Algorithms. Horwood, 2007

A. Niculescu-Mizil, R. Caruana: Predicting Good Probabilities With Supervised Learning. Proceedings of the 22nd International Conference on Machine Learning (ICML'05), 2005

See Also

CORElearn, predict.CoreModel.

Examples

# generate data
train <-classDataGen(noInst=200)
cal <-classDataGen(noInst=200)

# build random forests model with certain parameters
modelRF <- CoreModel(class~., train, model="rf",   selectionEstimator="MDL",minNodeWeight=5,rfNoTrees=100)
# prediction 
pred <- predict.CoreModel(modelRF, cal, rfPredictClass=FALSE)

# calibrate for a chosen class1 and method
class1<-1
calibrate(cal$class, pred$prob[,class1], class1=1, method="binning", noBins=5)