wrapPerformanceAnalyzer: Create a boostr compatible wrapper for a performance analyzer.

Description

Use provided metadata on a given performance analyzer to create a boostr compatible wrapper.

Usage

wrapPerformanceAnalyzer(analyzePerformance, analyzerInputPreds = "prediction", analyzerInputResponse = "response", analyzerInputOOBObs = "oobObs", .verbose = FALSE)

Arguments

analyzePerformance

a function to analyze the performance of an estimator

analyzerInputPreds

a string indicating the name of the argument in analyzePerformance's signature that represents the estimator's predictions.

analyzerInputResponse

a string indicating the name of the argument in analyzePerformance's signature that represents the true response associated with the estimator's predictions.

analyzerInputOOBObs

a string indiciating the name of the argument in analyzePerformance's signature that represents the vector of indices indicating which observations were out-of-bag.

.verbose

a boolean indicating if warnings should be displayed or not.

Value

A function (wrapper around analyzePerformance) which is also a 'performanceAnalyzer' object. The function's signature is (prediction, response, oobObs, ...) and it's output preserves the output of analyzePerformance. Hence, the wrapper is a boostr compatible performance analyzer.

Performance Analyzers

Any function which can accept an estimator's predictions, as well as the true responses can be used as a "performance analyzer" in boost. That is, if the signature of a function can be transformed to (predictions, responses, ...), then wrapPerformanceAnalyzer can be run on the function, and the results can be used by boostBackend. The output of the performance analyzer is entirely preserved, and can be accessed by running estimatorPerformance on the resulting 'boostr' object. At every iteration of the ensemble building phase, boostBackend passes performance Analysis:

the newly built estimator's predicted responses for each row in data.
the true response for each row in data.
the indices of the observations in data that were not included in the sample of data that went into creating the estimator. This variable is passed in as oobObs.
whatever other named arguments were passed from boost through the ...'s.

Hence, an analyzer can accept all three values (and then some) to perform whatever analysis is desired on an individual estimator. For example, the stock performance analyzer for classification, defaultOOBPerformanceAnalysis, has signature (prediction, response, oobObs) and calculates an individual, out-of-bag, misclassification vector, the overall out-of-bag error rate, and the confusion matrix for a particular estimator, given the information passed to it.

Details

Since "performance" is a subjective thing, the requirements for a function to be wrappable by wrapPerformanceAnalyzer are that they accept predictions, true responses, and a vector of indices for out-of-bag observations. After each iteration of the ensemble building phase in boostBackend, these three objects are fed to a performance analyzer. The output of the performance analyze is stored in the estimatorPerformance attribute of the object returned by boostBackend.