race (version 0.1.59)

race: Racing methods for the selection of the best

Description

Implementation of some racing methods for the empirical selection of the best. If the R package rpvm is installed (and PVM is available, properly configured, and initialized), the evaluation of the candidates are performed in parallel on different hosts.

Usage

race(wrapper.file, maxExp=0, stat.test=c("friedman","t.bonferroni","t.holm","t.none"), conf.level=0.95, first.test=5, interactive=TRUE, log.file="", no.slaves=0,...)

Arguments

wrapper.file

The name of a file containing the definition of the functions to be provided by the user: i.e. race.wrapper and race.info. The file wrapper.file might also define the functions race.init and race.describe.

maxExp

Maximum number of experiments (i.e. evaluations of the function race.wrapper) that are allowed before selecting the best candidate. If maxExp=0, no limit is imposed... very unrealistic in practice.

stat.test

Statistical test to be used for discarding inferior candidates.

conf.level

The confidence level to be used for the statistical test.

first.test

The first test for discarding inferior candidates is performed only when all candidates have been evaluated on a minimum number of tasks equal to first.test.

interactive

If TRUE, print a progress report on the standard output.

log.file

File for saving periodically the state of the race.

no.slaves

When running under PVM, no.slaves specify the number of slaves to be spawned. If no.slave=0 PVM is not used and all experiments are performed on the local host.

...

All extra parameters are passed to the function race.init defined by the user in the file wrapper.file.

Value

race is a list containing the following components:

precis

A string describing the race for documentation purposes.

results

A matrix containing in position [i,j] the result obtained by candidate j on task i.

no.candidates

Number of candidates at the beginning of the race.

no.tasks

Number of tasks on which the selection was based.

no.subtasks

Number of subtasks composing each tasks. Default=1

no.experiments

Number of times that the function race.wrapper had to be call in order to select the best.

no.alive

Number of candidates that completed the race, that is, number of candidates that had not been discarded at the moment in which the race was stopped.

alive

List of the candidates that completed the race: no sufficient evidence was gathered, give that the test stat.test is adopted, for stating that these candidates are worse than the selected best.

alive.inTime

Number of candidates in the race after each time step.

best

The candidate selected in the race.

mean.best

The average result of the best on the tasks considered.

description.best

An object describing the selected candidate.

timestamp.start

Time stamp of the beginning of the race.

timestamp.end

Time stamp of the end of the race.

Details

This package implemets some racing procedures for selecting from a set of candidate the one that is able to yield the best performance on a given set of tasks. The time available for selecting the best candidate is limited and, therefore, a brute-force approach is unfeasible. The algorithm implemented in this package sequentially evaluates the set of candidatas on the available tasks while discards bad candidates as soon as statistically sufficient evidence is gathered against them. The elimination of inferior candidates, speeds up the procedure and allows a more reliable evaluation of the promising ones.

References

O. Maron and A.W. Moore (1994) Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation. Advances in Neural Information Processing Systems 6, pp. 59--66. Morgan Kaufmann.

A.W. Moore and M.S. Lee (1994) Efficient Algorithms for Minimizing Cross Validation Error. International Conference on Machine Learning, pp. 190--198. Morgan Kaufmann.

O. Maron and A.W. Moore (1997) The Racing Algorithm: Model Selection for Lazy Learners. Artificial Intelligence Review, 11(1--5), pp. 193--225. M. Birattari, T. Stuetzle, L. Paquete, and K. Varrentrapp (2002) A Racing Algorithm for Configuring Metaheuristics. GECCO 2002: Genetic and Evolutionary Computation Conference, pp. 11--18. Morgan Kaufmann. M. Birattari (2004) The Problem of Tuning Metaheuristics as Seen from a Machine Learning Perspective. PhD Thesis, Universite' Libre de Bruxelles, Brussels, Belgium.

See Also

race.wrapper, race.info, race.init, race.describe

Examples

# The wrapper and init functions for this example are defined in the
# file examples/example-wrapper.R in the installation directory of the
# package.  Please, have a look at such file before implementing your
# own wrapper.
# This example require the package `nnet'
if (require(nnet)&&require(datasets)){
  example.wrapper<-file.path(system.file(package="race"),
                           "examples","example-wrapper.R")
  # Run the race
  race(example.wrapper)

  # If the package `rpvm' is installed on your system and if PVM is
  # properly installed and configured, you can try the following:
  #race(example.wrapper,no.slaves=6)
}