irace: An implementation in R of Iterated Race

Manuel López-Ibáñez
Jérémie Dubois-Lacoste
Contact email: irace@iridia.ulb.ac.be

Requisites

• R (http://www.r-project.org) is required for running irace, but you don't need to know the R language to use it. Versions that work: >= 2.14.0 (See Installing R below for a quick install guide)

• If you wish to use the command-line wrappers irace and parallel-irace, they require GNU Bash.

Installing R

The official instructions are available at http://cran.r-project.org/doc/manuals/r-release/R-admin.html

We give below a quick installation guide that will work in most cases.

GNU/Linux

You should install R from your package manager. On a Debian/Ubuntu system it will be something like:

$ sudo apt-get install r-base

Once R is installed, you can launch R from the Terminal and from the R prompt install the irace package. See instructions below.

OS X

You can install R directly from a CRAN mirror (the link of the Belgian one is http://cran.freestatistics.org/bin/macosx/)

Alternatively, if you use homebrew, you can just brew the R formula from the science tap (unfortunately it does not come already bottled so you need to have Xcode installed to compile it):

$ brew tap homebrew/science
$ brew install r

Once R is installed, you can launch R from the Terminal (or from your Applications), and from the R prompt install the irace package. See instructions below.

Windows

You can install R from a CRAN mirror (the link for the Belgian one is http://cran.freestatistics.org/bin/windows/)

Once R is installed, you can launch the R console and install the irace package from it. See instructions below.

Installing the irace package

Install the irace R package on your computer. There are two methods:

1. Install within R (automatic download):

    $ R
    > install.packages("irace")

   select a mirror close to you, and test the installation with

    > library(irace)
    > CTRL+d

2. Manually download the package from CRAN and invoke at the command-line:

    $ R CMD INSTALL <package>

   where <package> is one of the three versions available: .tar.gz (Unix/BSD/GNU/Linux), .tgz (MacOS X), or .zip (Windows).

If the package fails to install because of insufficient permissions, you need to force a local installation by doing:

$ mkdir ~/R
$ R CMD INSTALL --library=~/R irace.tar.gz
$ export R_LIBS=~/R:${R_LIBS}

Once installed, test that it is working by doing:

$ R
> library(irace)
> system.file(package="irace")
[1] "~/R/irace"

GNU/Linux and OS X

The last command tells you the installation directory of irace. Save that path to a variable, and add it to your .bash_profile, .bashrc or .profile:

export IRACE_HOME=~/R/irace/ # Path given by system.file(package="irace")
export PATH=${IRACE_HOME}/bin/:$PATH
# export R_LIBS=~/R:${R_LIBS} # Only if local installation was forced

After adding this and opening a new terminal, you should be able to invoke irace as follows:

$ irace --help

Windows

Unfortunately, the command-line wrapper does not work in Windows. To launch irace, you need to open the R console and execute:

R> library(irace)
R> irace.cmdline("--help")

Usage

1. Create a directory for storing the configuration of the tuning

    $ mkdir ~/tuning
    $ cd ~/tuning

2. Copy the template and example files to the tuning directory

    $ cp $IRACE_HOME/templates/*.tmpl .

   where $IRACE_HOME is the path to the installation directory of irace. It can be obtained by doing:

    $ R
    > library(irace)
    > system.file(package="irace")

3. For each template in your tuning directory, remove the .tmpl suffix, and modify them following the instructions in each file. In particular,

   • tune-main.tmpl should be adjusted depending on your usage (local, cluster, etc).
   • The scripts hook-run, hook-evaluate (if you need it at all) and tune-main should be executable. The output of hook-run (or hook-evaluate if you use a separate evaluation step) is minimized by default. If you wish to maximize it, just multiply the value by -1 within the script.
   • In tune-conf, uncomment and assign only the parameters for which you need a value different than the default one.

   There are examples in $IRACE_HOME/examples/.

4. Put the instances in ~/tuning/Instances/. In addition, you can create a file that specifies which instances from that directory should be run and which instance-specific parameters to use. See tune-conf.tmpl and instances-list.tmpl for examples. The command irace will not attempt to create the execution directory (execDir), so it must exist before calling irace. The default execDir is the current directory.

5. Calling the command:

    $ cd ~/tuning/ && $IRACE_HOME/bin/irace

   performs one run of Iterated Race. See the output of irace --help for additional configuration parameters. Command-line parameters override the configuration specified in the tune-conf file.

Many tuning runs in parallel

For executing several repetitions of irace in parallel, call the program

$ cd ~/tuning/ && $IRACE_HOME/bin/parallel-irace N

where N is the number of repetitions. By default, the execution directory of each run of irace will be set to ./TUNE-dd, where dd is a number padded with zeroes.

Be careful, parallel-irace will create these directories from scratch, deleting them first if they already exist.

Check the help of parallel-irace by running it without parameters.

Parallelize one tuning

A single run of irace can be done much faster by executing the calls to hookRun (the runs of the algorithm being tuned) in parallel. There are three ways to parallelize a single run of irace.

1. Parallel processes: The option --parallel N will use the parallel package to launch locally up to N calls of hookRun in parallel.

2. MPI: The option --mpi 1 --parallel N will use the Rmpi package to launch N slaves + 1 master, in order to execute N calls of hookRun in parallel. The user is responsible to set up MPI correctly.

   An example script for using MPI mode in an SGE cluster is given at $IRACE_HOME/examples/mpi/.

3. SGE cluster: This mode requires Grid Engine commands qsub and qstat. The command qsub should return a message that contains the string: Your job JOBID. The command qstat -j JOBID should return nonzero if JOBID has finished, otherwise it should return zero.

   The option --sge-cluster 1 will launch as many calls of hookRun as possible and use qstat to wait for cluster jobs. The user must call qsub from hookRun with the appropriate configuration for their cluster, otherwise hookRun will not submit jobs to the cluster. In this mode, irace must run in the submission node, and hence, qsub should not be used to invoke irace (either directly or through tune-main). You also need to create a separate hookEvaluate script to parse the results of the hookRun and return them to irace. See the examples in $IRACE_HOME/examples/sge-cluster/.

Frequently Asked Questions

Is irace minimizing or maximizing the output of my algorithm?

By default, irace considers that the value returned by hook-run (or by hook-evaluate, if used) should be minimized. In case of a maximization problem, one can simply multiply the value by -1 before returning it to irace. This is done, for example, when maximizing the hypervolume (see the last lines in $IRACE_HOME/examples/hypervolume/hook-evaluate).

Is it possible to configure a MATLAB algorithm with irace?

Definitely. There are two main ways to achieve this:

1. Edit the hook-run script to call MATLAB in a non-interactive way. See the MATLAB documentation, or the following links: http://stackoverflow.com/questions/1518072/suppress-start-message-of-matlab/ http://stackoverflow.com/questions/4611195/how-to-call-matlab-from-command-line-and-print-to-stdout-before-exiting

   You would need to pass the parameter received by hook-run to your MATLAB script: http://www.mathworks.nl/support/solutions/en/data/1-1BS5S/?solution=1-1BS5S

2. Call MATLAB code directly from R using R.matlab package. This is a better option if you are experienced in R. Define hookRun as an R function instead of a path to a script. The function should call your MATLAB code with appropriate parameters.

My program works perfectly on its own, but not when running under irace. Is irace broken?

Every time this was reported, it was a difficult-to-reproduce bug in the program, not in irace. We recommend that in hook-run, you use valgrind to run your program. That is, if you program is called like:

$EXE ${FIXED_PARAMS} -i $INSTANCE ${CAND_PARAMS} 1> ${STDOUT} 2> ${STDERR}

then replace that line with:

valgrind --error-exitcode=1 $EXE ${FIXED_PARAMS} -i $INSTANCE ${CAND_PARAMS} 1> ${STDOUT} 2> ${STDERR}

If there are bugs in your program, they will appear in ${STDERR}, thus do not delete those files.

My program may be buggy and run into an infinite loop. Is it possible to set a maximum timeout?

We are not aware of any way to achieve this using R. However, in GNU/Linux, it is easy to implement by using the timeout command in hook-run when invoking your program.

Contact

If you find that you need to modify anything else, something is not clear or if you have problems, please contact irace@iridia.ulb.ac.be