DAAGxtras (version 0.8-4)

compareModels: Compare accuracy of alternative classification methods

Description

Compare, between models, probabilities that the models assign to membership in the correct group or class. Probabilites should be estimated from cross-validation or from bootstrap out-of-bag data or preferably for test data that are completely separate from the data used to dervive the model.

Usage

compareModels(groups = fgl$type, estprobs = list(lda = NULL, rf = NULL), gpnames = NULL, robust = TRUE, print = TRUE)

Arguments

groups

Factor that specifies the groups

estprobs

List whose elements (with names that identify the models) are matrices that give for each observation (row) estimated probabilities of membership for each of the groups (columns).

gpnames

Character: names for groups, if different from levels(groups)

robust

Logical, TRUE or FALSE

print

Logical. Should results be printed?

Value

modelAVS

Average accuracies for models

modelSE

Approximate average SE for comparing models

gpAVS

Average accuracies for groups

gpSE

Approximate average SE for comparing groups

obsEff

Effects assigned to individual observations

Details

The estimated probabilities are compared directly, under normal distribution assumptions. An effect is fitted for each observation, plus an effect for the method. Comparison on a logit scale may sometimes be preferable. An option to allow this is scheduled for incorporation in a later version.

Examples

library(MASS)
library(DAAG)
ldahat <- lda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
qdahat <- qda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
compareModels(groups=cuckoos$species, estprobs=list(lda=ldahat,
              qda=qdahat), robust=FALSE)
rfOUT <- try(require(randomForest, quietly=TRUE))
rfOUT.log <- is.logical(rfOUT)
if ((rfOUT.log==TRUE)&(rfOUT==TRUE)){
rfhat <- predict(randomForest(species ~ length+breadth, data=cuckoos),
                 type="prob")
compareModels(groups=cuckoos$species, estprobs=list(lda=ldahat,
              qda=qdahat, rf=rfhat), robust=FALSE)
}