parscores: Penalized average runtime score

Description

Calculates the penalized average runtime score which is commonly used for evaluating satisfiability solvers on a set of problems.

Usage

parscores(data, model, factor = 10, timeout)

Arguments

data

the data used to induce the model. The same as given to classify, classifyPairs, cluster or regression.

model

the algorithm selection model. Can be either a model returned by one of the model-building functions or a function that returns predictions such as vbs or the predictor function of a trained model.

factor

the penalization factor to use for non-successful choices. Default 10.

timeout

the timeout value to be multiplied by the penalization factor. If not specified, the maximum performance value of all algorithms on the entire data is used.

Value

A list of the penalized average runtimes.

Details

Returns the penalized average runtime performances of the respective chosen algorithm on each problem instance.

If feature costs have been given, the cost of the used features or feature groups is added to the performance of the chosen algorithm. The used features are determined by examining the the features member of data, not the model. If after that the performance value is above the timeout value, the timeout value multiplied by the factor is assumed.

Examples

Run this code

data(satsolvers)
folds = cvFolds(satsolvers)

model = classify(classifier=makeLearner("classif.J48"), data=folds)
sum(parscores(folds, model))

# use factor of 5 instead of 10.
sum(parscores(folds, model, 5))

# explicitly specify timeout.
sum(parscores(folds, model, timeout = 3600))