{"Package":"mlr3benchmark","Title":"Analysis and Visualisation of Benchmark Experiments","Version":"0.1.1","Authors@R":"\nc(person(given = \"Sonabend\",\nfamily = \"Raphael\",\nrole = c(\"cre\", \"aut\"),\nemail = \"raphael.sonabend.15@ucl.ac.uk\",\ncomment = c(ORCID = \"0000-0001-9225-4654\")),\nperson(given = \"Florian\",\nfamily = \"Pfisterer\",\nrole = \"aut\",\nemail = \"pfistererf@googlemail.com\",\ncomment = c(ORCID = \"0000-0001-8867-762X\")),\nperson(given = \"Michel\",\nfamily = \"Lang\",\nrole = \"ctb\",\nemail = \"michellang@gmail.com\",\ncomment = c(ORCID = \"0000-0001-9754-0393\")),\nperson(given = \"Bernd\",\nfamily = \"Bischl\",\nrole = \"ctb\",\nemail = \"bernd_bischl@gmx.net\",\ncomment = c(ORCID = \"0000-0001-6002-6980\")))","Description":"Implements methods for post-hoc analysis and\nvisualisation of benchmark experiments, for 'mlr3' and beyond.","License":"LGPL-3","URL":"https://mlr3benchmark.mlr-org.com,\nhttps://github.com/mlr-org/mlr3benchmark","BugReports":"https://github.com/mlr-org/mlr3benchmark/issues","Depends":"R (>= 3.1.0)","Imports":"checkmate, data.table, ggplot2, mlr3misc, R6","Suggests":"mlr3, mlr3learners, PMCMR, rpart, testthat, xgboost","Encoding":"UTF-8","NeedsCompilation":"no","RoxygenNote":"7.1.1","Packaged":"2020-12-16 09:57:16 UTC; raphael","Author":"Sonabend Raphael [cre, aut] (),\nFlorian Pfisterer [aut] (),\nMichel Lang [ctb] (),\nBernd Bischl [ctb] ()","Maintainer":"Sonabend Raphael ","Repository":"CRAN","Date/Publication":"2020-12-16 12:20:05 UTC","repoType":"cran","tarballUrl":"ftp://cran.r-project.org/pub/R/src/contrib/mlr3benchmark_0.1.1.tar.gz","jsonAuthors":[{"name":"Sonabend Raphael","email":"raphael.sonabend.15@ucl.ac.uk","maintainer":true},{"name":"Florian Pfisterer","email":"pfistererf@googlemail.com","maintainer":false},{"name":"Michel Lang","email":"michellang@gmail.com","maintainer":false},{"name":"Bernd Bischl","email":"bernd_bischl@gmx.net","maintainer":false}],"readme":"\n# mlr3benchmark\n\nAnalysis and tools for benchmarking in\n[mlr3](https://github.com/mlr-org/mlr3).\n\n\n\n[![tic](https://github.com/mlr-org/mlr3benchmark/workflows/tic/badge.svg?branch=master)](https://github.com/mlr-org/mlr3benchmark/actions)\n[![CRAN\nStatus](https://www.r-pkg.org/badges/version-ago/mlr3benchmark)](https://cran.r-project.org/package=mlr3benchmark)\n[![CRAN\nchecks](https://cranchecks.info/badges/worst/mlr3benchmark)](https://cran.r-project.org/web/checks/check_results_mlr3benchmark.html)\n\n[![CodeFactor](https://www.codefactor.io/repository/github/mlr-org/mlr3benchmark/badge)](https://www.codefactor.io/repository/github/mlr-org/mlr3benchmark)\n[![Codecov test\ncoverage](https://codecov.io/gh/mlr-org/mlr3benchmark/branch/master/graph/badge.svg)](https://codecov.io/gh/mlr-org/mlr3benchmark?branch=master)\n\n[![StackOverflow](https://img.shields.io/badge/stackoverflow-mlr3-orange.svg)](https://stackoverflow.com/questions/tagged/mlr3)\n[![Mattermost](https://img.shields.io/badge/chat-mattermost-orange.svg)](https://lmmisld-lmu-stats-slds.srv.mwn.de/mlr_invite/)\n\n\n## What is mlr3benchmark?\n\nDo you have a large benchmark experiment with many tasks, learners, and\nmeasures, and don’t know where to begin with analysis? Do you want to\nperform a complete quantitative analysis of benchmark results to\ndetermine which learner truly is the ‘best’? Do you want to visualise\ncomplex results for benchmark experiments in one line of code?\n\nThen **mlr3benchmark** is the answer, or at least will be once it’s\nfinished maturing.\n\n**mlr3benchmark** enables fast and efficient analysis of benchmark\nexperiments in just a few lines of code. As long as you can coerce your\nresults into a format fitting our classes (which have very few\nrequirements), then you can perform your benchmark analysis with\n**mlr3benchmark**.\n\n## Installation\n\nInstall the last release from CRAN:\n\n``` r\ninstall.packages(\"mlr3benchmark\")\n```\n\nInstall the development version from GitHub:\n\n``` r\nremotes::install_github(\"mlr-org/mlr3benchmark\")\n```\n\n## Feature Overview\n\nCurrently **mlr3benchmark** only supports analysis of multiple learners\nover multiple tasks. The current implemented features are best\ndemonstrated by example\\!\n\nFirst we run a mlr3 benchmark experiment:\n\n``` r\nlibrary(mlr3)\nlibrary(mlr3learners)\nlibrary(ggplot2)\nset.seed(1)\n\ntask = tsks(c(\"iris\", \"sonar\", \"wine\", \"zoo\"))\nlearns = lrns(c(\"classif.featureless\", \"classif.rpart\", \"classif.xgboost\"))\nbm = benchmark(benchmark_grid(task, learns, rsmp(\"cv\", folds = 3)))\n```\n\nNow we create a `BenchmarkAggr` object for our analysis, these objects\nstore measure results after being *aggregated* over all resamplings:\n\n``` r\n# these measures are the same but we'll continue for the example\naggr = bm$aggregate(msrs(c(\"classif.acc\", \"classif.ce\")))\nba = BenchmarkAggr$new(aggr)\nba\n```\n\n ## of 12 rows with 4 tasks, 3 learners and 2 measures\n ## task_id learner_id acc ce\n ## 1: iris featureless 0.2800000 0.72000000\n ## 2: iris rpart 0.9466667 0.05333333\n ## 3: iris xgboost 0.9600000 0.04000000\n ## 4: sonar featureless 0.5334023 0.46659765\n ## 5: sonar rpart 0.6537612 0.34623879\n ## 6: sonar xgboost 0.6394755 0.36052450\n ## 7: wine featureless 0.3990584 0.60094162\n ## 8: wine rpart 0.8652542 0.13474576\n ## 9: wine xgboost 0.9048023 0.09519774\n ## 10: zoo featureless 0.4058229 0.59417706\n ## 11: zoo rpart 0.8309566 0.16904337\n ## 12: zoo xgboost 0.9099822 0.09001783\n\nNow we can begin our analysis\\! In **mlr3benchmark**, analysis of\nmultiple learners over multiple independent tasks follows the guidelines\nof Demsar (2006). So we begin by checking if the global Friedman test is\nsignificant: is there are a significant difference in the rankings of\nthe learners over all the tasks?\n\n``` r\nba$friedman_test()\n```\n\n ## X2 df p.value p.signif\n ## acc 6.5 2 0.03877421 *\n ## ce 6.5 2 0.03877421 *\n\nBoth measures are significant, so now we can proceed with the post-hoc\ntests. Now comparing each learner to each other with post-hoc\nFriedman-Nemenyi tests:\n\n``` r\nba$friedman_posthoc(meas = \"acc\")\n```\n\n ## \n ## Pairwise comparisons using Nemenyi multiple comparison test \n ## with q approximation for unreplicated blocked data \n ## \n ## data: acc and learner_id and task_id \n ## \n ## featureless rpart\n ## rpart 0.181 - \n ## xgboost 0.036 0.759\n ## \n ## P value adjustment method: none\n\nThe results tell us that xgboost is significantly different from the\nfeatureless model, but all other comparisons are non-significant. This\ndoesn’t tell us *which* of xgboost and featureless is better though, the\nmost detailed information is given in a critical difference diagram,\nnote we include `minimize = FALSE` as accuracy should be maximised:\n\n``` r\nautoplot(ba, type = \"cd\", meas = \"acc\", minimize = FALSE)\n```\n\n![](man/figures/README-unnamed-chunk-7-1.png)\n\nWe read the diagram from left to right, so that learners to the left\nhave the highest rank and are the best performing, and decrease going\nright. The thick horizontal lines connect learners that are *not*\nsignificantly difference in ranked performance, so this tells us:\n\n1. xgboost is significantly better than featureless\n2. xgboost is not significantly better than rpart\n3. rpart is not significantly better than featureless\n\nNow we visualise two much simpler plots which display similar\ninformation, the first is the mean and standard error of the results\nacross all tasks, the second is a boxplot across all tasks:\n\n``` r\nautoplot(ba, meas = \"acc\")\n```\n\n![](man/figures/README-unnamed-chunk-8-1.png)\n\n``` r\nautoplot(ba, type = \"box\", meas = \"acc\")\n```\n\n![](man/figures/README-unnamed-chunk-8-2.png)\n\nWe conclude that xgboost is significantly better than the baseline but\nnot significantly better than the decision tree but the decision tree is\nnot significantly better than the baseline, so we will recommend xgboost\nfor now.\n\nThe analysis is complete\\!\n\n## Roadmap\n\n**mlr3benchmark** is in its early stages and the interface is still\nmaturing, near-future updates will include:\n\n - Extending `BenchmarkAggr` to non-independent tasks\n - Extending `BenchmarkAggr` to single tasks\n - Adding `BenchmarkScore` for non-aggregated measures,\n e.g. observation-level scores\n - Bayesian methods for analysis\n\n## Bugs, Questions, Feedback\n\n**mlr3benchmark** is a free and open source software project that\nencourages participation and feedback. If you have any issues,\nquestions, suggestions or feedback, please do not hesitate to open an\n“issue” about it on the [GitHub\npage](https://github.com/mlr-org/mlr3benchmark/issues)\\! In case of\nproblems / bugs, it is often helpful if you provide a “minimum working\nexample” that showcases the behaviour (but don’t worry about this if the\nbug is obvious).","jobInfo":{"package":"mlr3benchmark","version":"0.1.1","parsingStatus":"success","parserVersion":1,"parsedAt":"2021-04-19T17:29:53+0000"}}

Description

Usage

Arguments