Learn R Programming

IOHanalyzer

The performance analyzer for Iterative Optimization Heuristics (IOHs).

It provides:

  • a web-based graphical user interface (GUI) to analyze and visualize the empirical performance of IOHs,
  • interactive plotting,
  • statistical evaluation,
  • report generation, and
  • a command-line interface (CLI) for the R console allowing for fine-grained controls.

It is built mainly on:

It is available through:

Online Service

A free server https://iohanalyzer.liacs.nl running the stable version of IOHanalyzer is hosted in Leiden Institute of Advanced Computer Science. You're welcome to check it out!

Installation

Software dependency

  • [mandatory] R As IOHanalyzer is written as a R package, the R environment has to be installed first. The binary file and installation manual for R can be found here https://cran.r-project.org/.
  • [optional] either kaleido (recommended) or orca (will be depricated) is required to download plotly figures. Please see Kaleido or Orca for their respective installation instructions.

Stable version

Please start up an R console and install the stable version as:

R> install.packages('IOHanalyzer')

which is maintained on CRAN (Comprehensive R Archive Network).

Lastest version

The lastest development is always hosted on Github. In case you'd like to try out this version, the R package devtool is needed:

R> install.packages('devtools')
R> devtools::install_github('IOHprofiler/IOHanalyzer')

Development version

If you want to run the version on which you develop:

R> devtools::install_git("/path/to/your/IOHanalyzer/git/repo")

Runinng the Web Interface locally

The IOHanalyzer package can be loaded using the following commands:

R> library('IOHanalyzer')
R> runServer()

It should open a browser on the localhost server, using a random port number. Of course, you could also specify the port number directly:

R> runServer(port = 1234)

Have fun! For the complete reference on usage, please check out our Wiki page.

Host it online

We provide a docker file for deploying IOHanalyzer on the server. Please see https://github.com/IOHprofiler/IOHanalyzer-docker for details.

Supported Data Format

Specific formats are required to load your benchmark data to IOHanalyzer. If your data sets are generated in the format of

then you just need to compress the data folder obtained from the experiment into a zip file and uploaded it. However, you are encouraged to convert your own benchmark data to the format regulated here!. The supported data format is specified in this page. Please follow the instruction there to convert your data sets.

Our Team

  • Hao Wang, Sorbonne Université, LIP6, France.
  • Diederick Vermetten, Leiden Institute of Advanced Computer Science, The Netherlands.
  • Furong Ye, Leiden Institute of Advanced Computer Science, The Netherlands.
  • Ofer M. Shir, Tel-Hai College, Israel.
  • Carola Doerr, Sorbonne Université, CNRS, LIP6, France.
  • Thomas Bäck, Leiden Institute of Advanced Computer Science, The Netherlands.

When using IOHprofiler and parts thereof, please kindly cite this work as

Hao Wang, Diederick Vermettern, Furong Ye, Carola Doerr and Thomas Bäck: IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic, arXiv e-prints:2007.03953, 2020.

@ARTICLE{IOHprofiler,
  author = {Hao Wang and Diederick Vermettern and Furong Ye and Carola Doerr and Thomas B{\"a}ck},
  title = {{IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic}},
  journal = {arXiv e-prints:2007.03953},
  archivePrefix = "arXiv",
  eprint = {2007.03953},
  year = 2020,
  month = July,
  keywords = {Computer Science - Neural and Evolutionary Computing},
  url = {https://arxiv.org/abs/2007.03953}
}

License

This application is governed by the BSD 3-Clause license.

BSD 3-Clause License

Copyright (c) 2018, All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  • Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Copy Link

Version

Install

install.packages('IOHanalyzer')

Monthly Downloads

417

Version

0.1.8.10

License

BSD_3_clause + file LICENSE

Issues

Pull Requests

Stars

Forks

Maintainer

Diederick Vermetten

Last Published

March 1st, 2024

Functions in IOHanalyzer (0.1.8.10)

Plot.FV.Multi_Func

Plot FV-plots for multiple functions or dimensions
Plot.RT.Parameters

Plot the parameter values recorded in a DataSetList (aligned by funcion value)
cat.DataSet

S3 generic cat operator for DataSet
Plot.RT.ECDF_Multi_Func

Plot the aggregated empirical cumulative distriburtion as a function of the running times of a DataSetList. Aggregated over multiple functions or dimensions.
change_id

Add unique identifiers to each DataSet in the provided DataSetList based on static attributes
Plot.cumulative_difference_plot

Plot the cumulative difference plot given a DataSetList.
Plot.RT.ECDF_Per_Target

Plot the empirical cumulative distriburtion as a function of the running times of a DataSetList at certain target function values
Plot.Stats.Significance_Heatmap

Plot a heatmap showing the statistically different algorithms
Plot.RT.Single_Func

Plot lineplot of the ERTs of a DataSetList
Plot.RT.Multi_Func

Plot ERT-plots for multiple functions or dimensions
c.DataSet

S3 concatenation function for DataSet
clean_DataSetList

Clean DataSetList object by concatenating DataSets
generate_data.CDP

Generate data for the cumulative difference plot.
Plot.Stats.Glicko2_Candlestick

Create a candlestick plot of Glicko2-rankings
Plot.RT.PMF

Plot probablity mass function of the runtimes of a DataSetList at a certain target function value
c.DataSetList

S3 concatenation function for DataSetList
as.character.DataSet

S3 generic as.character operator for DataSet
fast_RT_samples

Function to get just the RT samples needed, without any formatting to improve speed
generate_data.AUC

Generate dataframe containing the AUC for any ECDF-curves
generate_data.Aggr

Generate dataframe of a single function/dimension pair
Plot.RT.Histogram

Plot histograms of the runtimes of a DataSetList at a certain target function value
Plot.RT.ECDF_Single_Func

Plot the aggregated empirical cumulative distriburtion as a function of the running times of a DataSetList.
bootstrap_RT

Bootstrapping for running time samples
dsl

Example DataSetList used in tests / examples
Plot.Stats.Significance_Graph

Plot a network graph showing the statistically different algorithms
generate_data.Parameters

Generate dataframe of a single function/dimension pair
generate_data.Single_Function

Generate dataframe of a single function/dimension pair
SP

Estimator 'SP' for the Expected Running Time (ERT)
generate_data.Parameter_correlation

Generate dataframe of exactly 2 parameters, matched by running time
get_ECDF_targets

Generation of default ECDF-targets
generate_data.hist

Generate dataframe of a single function/dimension pair
generate_data.EAF_diff_Approximate

Generate EAF-differences between each function and the remaining portfolio
get_marg_contrib_ecdf

Get the marginal contribution of an algorithm to a portfolio
generate_data.ECDF_raw

Generate dataframe of a the unaggregated values of individual algorithms. Stripped-down version of
get_FV_sample

Get Funtion Value Samples
check_format

Check the format of data
check_dsc_configured

Verify that the credentials for DSCtool have been set
generate_data.ECDF_From_EAF

Generate dataframe consisting of the ECDF-equivalent based on the EAF
get_static_attributes

Get all attributes which can be used to subset a DataSetList
get_maxRT

Get the maximal running time
arrange

S3 sort function for DataSetList
get_FV_overview

Get Function Value condensed overview
get_FV

Get function value matrix of the used dataset.
get_PAR_summary

Get Parameter Value Summary
get_RT

Get runtime matrix of the used dataset.
generate_data.ECDF

Generate dataframe of a single function/dimension pair
get_dsc_omnibus

Perform omnibus statistical tests on the matrix of rankings from the DSCtool api
dsl_large

Larger example DataSetList used in tests / examples
generate_data.PMF

Generate dataframe of a single function/dimension pair for creating PDF or PMF plots
generate_data.Heatmaps

Nevergrad-dashboard based algorithm comparison
==.DataSet

S3 generic == operator for DataSets
generate_data.EAF

Generate dataframe consisting of the levelsets of the EAF
generate_data.EAF_Difference

Generate differences between two EAFs
get_shapley_values

Get the shapley-values of a portfolio of algorithms
get_dsc_posthoc

Perform post-hoc processing on data from DSCtool
print.DataSet

S3 generic print operator for DataSet
get_ERT

Get Expected RunTime
get_dim

Get all dimensions present in a DataSetList
get_default_ECDF_targets

Generate ECDF targets for a DataSetList
print.DataSetList

S3 print function for DataSetList
read_pure_csv

Read Nevergrad data
register_DSC

Register an account to the DSCtool API
limit.data

Reduce the size of the data set by evenly subsampling the records
read_index_file

Read .info files and extract information
glicko2_ranking

Glicko2 raning of algorithms
get_static_attribute_values

Get all options for a specific attribute which can be used to subset a DataSetList
read_IOH_v1plus

Read Nevergrad data
get_target_dt

Generate datatables of runtime or function value targets for a DataSetList
get_RT_sample

Get RunTime Sample
get_funcName

Get all function names present in a DataSetList
get_RT_overview

Get Runtime Value condensed overview
set_DSC_credentials

Register an account to the DSCtool API
get_funvals

Get all function values present in a DataSetList
get_overview

Get condensed overview of datasets
plot_eaf_differences

Create EAF-difference contour plots
seq_RT

Function for generating sequences of runtime values
get_FV_summary

Get Function Value Summary
get_color_scheme_dt

Get datatable of current color (and linestyle) scheme to file
get_color_scheme

Get colors according to the current colorScheme of the IOHanalyzer
get_parId

Get all parameter ids present in a DataSetList
seq_FV

Function for generating sequences of function values
subset.DataSet

S3 subset function for DataSet
[.DataSetList

S3 extraction function for DataSetList
get_PAR_name

Get the parameter names of the algorithm
get_dsc_rank

Get the matrix of rankings using the DSCtool api for a DataSetList
get_ontology_var

Get the list of available options for data from the OPTION ontology
get_RT_summary

Get RunTime Summary
get_algId

Get all algorithm ids present in a DataSetList
get_id

Get condensed overview of datasets
get_PAR_sample

Get Parameter Value Samples
get_line_style

Get line styles according to the current styleScheme of the IOHanalyzer
get_ontology_data

Get the list of available options for data from the OPTION ontology
get_funcId

Get all function ids present in a DataSetList
plot_general_data

General function for plotting within IOHanalyzer
set_color_scheme

Set the colorScheme of the IOHanalyzer plots
get_position_dsl

Extract the position information from a datasetlist object
pairwise.test

Performs a pairwise Kolmogorov-Smirnov test on the bootstrapped running times among a data set
runServer

Create a shiny-server GUI to interactively use the IOHanalyzer
plot_eaf_data

Create EAF-based polygon plots
save_table

Save DataTable in multiple formats
scan_index_file

Scan *.info files for IOHProfiler or COCO
summary.DataSetList

S3 summary function for DataSetList
save_plotly

Save plotly figure in multiple format
mean_FVs

Get the expected function-values for all DataSets in a DataSetList at certain runtimes
get_runtimes

Get all runtime values present in a DataSetList
subset.DataSetList

Filter a DataSetList by some criteria
summary.DataSet

S3 generic summary operator for DataSet
max_ERTs

Get the ERT-values for all DataSets in a DataSetList at certain targets
AUC

Area Under Curve (Empirical Cumulative Dsitribution Function)
IOH_plot_ly_default

Template for creating plots in the IOHanalyzer-style
Plot.FV.Aggregated

Plot expected function value-based comparison over multiple functions or dimensions
IOHanlyzer-deprecated

Deprecated function in package IOHanalyzer
DataSet

Constructor of S3 class 'DataSet'
Plot.Comparison.Heatmap

Plot a heatmap according to the specifications from the Nevergrad dashboard
IOHanalyzer

IOHanalyzer: Data Analysis Part of IOHprofiler
ECDF

Empirical Cumulative Dsitribution Function of Runtime of a single data set
DataSetList

S3 constructor of the 'DataSetList'
Plot.RT.Aggregated

Plot ERT-based comparison over multiple functions or dimensions
Plot.FV.PDF

Plot probablity density function of the function values of a DataSetList at a certain target runtime
Plot.FV.ECDF_AUC

Radarplot of the area under the aggregated ECDF-curve of a DataSetList.
Plot.Performviz

Create the PerformViz plot
Plot.FV.Single_Func

Plot lineplot of the expected function values of a DataSetList
Plot.FV.Parameters

Plot the parameter values recorded in a DataSetList (aligned by budget)
Plot.FV.ECDF_Single_Func

Plot the aggregated empirical cumulative distriburtion as a function of the function values of a DataSetList.
Plot.FV.ECDF_Per_Target

Plot the empirical cumulative distriburtion as a function of the target values of a DataSetList at certain target runtimes
Plot.FV.Histogram

Plot histograms of the function values of a DataSetList at a certain target runtime
Plot.RT.ECDF_AUC

Radarplot of the area under the aggregated ECDF-curve of a DataSetList.