{"Package":"prcbench","Type":"Package","Title":"Testing Workbench for Precision-Recall Curves","Version":"1.1.8","Date":"2023-03-12","Authors@R":"c(\nperson(\"Takaya\", \"Saito\", email = \"takaya.saito@outlook.com\", role = c(\"aut\", \"cre\"),\ncomment = c(ORCID = \"0000-0002-0154-8452\")),\nperson(\"Marc\", \"Rehmsmeier\", email = \"marc.rehmsmeier@ii.uib.no\", role = c(\"aut\"),\ncomment = c(ORCID = \"0000-0002-5021-7721\")))","Description":"A testing workbench to evaluate tools that calculate precision-recall curves.\nSaito and Rehmsmeier (2015) .","URL":"https://evalclass.github.io/prcbench/,\nhttps://github.com/evalclass/prcbench","BugReports":"https://github.com/evalclass/prcbench/issues","Depends":"R (>= 3.2.3)","License":"GPL-3","Language":"en-US","LazyData":"TRUE","LinkingTo":"Rcpp","Imports":"Rcpp (>= 1.0.9), R6 (>= 2.1.1), assertthat (>= 0.1), grid,\ngridExtra (>= 2.0.0), graphics, ggplot2 (>= 2.1.0), methods,\nmemoise (>= 1.0.0), ROCR (>= 1.0-7), PRROC (>= 1.1), precrec\n(>= 0.1)","RoxygenNote":"7.2.3","Encoding":"UTF-8","Suggests":"microbenchmark (>= 1.4-2.1), rJava (>= 0.9-7), testthat (>=\n0.11.0), knitr (>= 1.11), rmarkdown (>= 0.8.1), vdiffr (>=\n1.0.0), patchwork (>= 1.1.2)","VignetteBuilder":"knitr","NeedsCompilation":"yes","Packaged":"2023-03-12 15:47:53 UTC; takaya","Author":"Takaya Saito [aut, cre] (),\nMarc Rehmsmeier [aut] ()","Maintainer":"Takaya Saito ","Repository":"CRAN","Date/Publication":"2023-03-12 16:10:09 UTC","repoType":"cran","tarballUrl":"ftp://cran.r-project.org/pub/R/src/contrib/prcbench_1.1.8.tar.gz","jsonAuthors":[{"name":"Takaya Saito","email":"takaya.saito@outlook.com","maintainer":true},{"name":"Marc Rehmsmeier","email":"marc.rehmsmeier@ii.uib.no","maintainer":false}],"readme":"\n# prcbench $\"\"$ \n\n[![R-CMD-check](https://github.com/evalclass/prcbench/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/evalclass/prcbench/actions/workflows/R-CMD-check.yaml)\n[![AppVeyor Build\nStatus](https://ci.appveyor.com/api/projects/status/github/evalclass/prcbench?branch=main&svg=true)](https://ci.appveyor.com/project/takayasaito/prcbench/)\n[![codecov.io](https://codecov.io/github/evalclass/prcbench/coverage.svg?branch=main)](https://codecov.io/github/evalclass/prcbench?branch=main)\n[![CodeFactor](https://www.codefactor.io/repository/github/evalclass/prcbench/badge)](https://www.codefactor.io/repository/github/evalclass/prcbench/)\n[![CRAN_Status_Badge](https://www.r-pkg.org/badges/version-ago/prcbench)](https://cran.r-project.org/package=prcbench)\n[![CRAN_Logs_Badge](https://cranlogs.r-pkg.org/badges/grand-total/prcbench)](https://cran.r-project.org/package=prcbench)\n\nThe aim of the `prcbench` package is to provide a testing workbench for\nevaluating precision-recall curves under various conditions. It contains\nintegrated interfaces for the following five tools. It also contains\npredefined test data sets.\n\n| Tool | Language | Link |\n|:--------------|:---------|:----------------------------------------------------------------------------------------------------------|\n| precrec | R | [Tool web site](https://evalclass.github.io/precrec/), [CRAN](https://cran.r-project.org/package=precrec) |\n| ROCR | R | [Tool web site](https://ipa-tys.github.io/ROCR/), [CRAN](https://cran.r-project.org/package=ROCR) |\n| PRROC | R | [CRAN](https://cran.r-project.org/package=PRROC) |\n| AUCCalculator | Java | [Tool web site](http://mark.goadrich.com/programs/AUC/) |\n| PerfMeas | R | [CRAN](https://cran.r-project.org/package=PerfMeas) |\n\n**Disclaimer**: `prcbench` was originally develop to help our\n[precrec](https://CRAN.R-project.org/package=precrec) library in order\nto provide fast and accurate calculations of precision-recall curves\nwith extra functionality.\n\n## Accuracy evaluation of precision-recall curves\n\n`prcbench` uses pre-defined test sets to help evaluate the accuracy of\nprecision-recall curves.\n\n1. `create_toolset`: creates objects of different tools for testing (5\n different tools)\n2. `create_testset`: selects pre-defined data sets (c1, c2, and c3)\n3. `run_evalcurve`: evaluates the selected tools on the simulation data\n4. `autoplot`: shows the results with `ggplot2` and `patchwork`\n\n``` r\n## Load library\nlibrary(prcbench)\n\n## Plot base points and the result of 5 tools on pre-defined test sets (c1, c2, and c3)\ntoolset <- create_toolset(c(\"precrec\", \"ROCR\", \"AUCCalculator\", \"PerfMeas\", \"PRROC\"))\ntestset <- create_testset(\"curve\", c(\"c1\", \"c2\", \"c3\"))\nscores1 <- run_evalcurve(testset, toolset)\nautoplot(scores1, ncol = 3, nrow = 2)\n```\n\n![](https://raw.githubusercontent.com/evalclass/prcbench/main/README_files/figure-gfm/fig1-1.png)\n\n## Running-time evaluation of precision-recall curves\n\n`prcbench` helps create simulation data to measure computational times\nof creating precision-recall curves.\n\n1. `create_toolset`: creates objects of different tools for testing\n2. `create_testset`: creates simulation data\n3. `run_benchmark`: evaluates the selected tools on the simulation data\n\n``` r\n## Load library\nlibrary(prcbench)\n\n## Run benchmark for auc5 (5 tools) on b10 (balanced 5 positives and 5 negatives)\ntoolset <- create_toolset(set_names = \"auc5\")\ntestset <- create_testset(\"bench\", \"b10\")\nres <- run_benchmark(testset, toolset)\n\nprint(res)\n```\n\n| testset | toolset | toolname | min | lq | mean | median | uq | max | neval |\n|:--------|:--------|:--------------|-----:|-----:|-----:|-------:|-----:|-----:|------:|\n| b10 | auc5 | AUCCalculator | 0.93 | 0.96 | 1.12 | 1.00 | 1.00 | 1.68 | 5 |\n| b10 | auc5 | PerfMeas | 0.06 | 0.06 | 0.08 | 0.06 | 0.07 | 0.17 | 5 |\n| b10 | auc5 | precrec | 3.40 | 3.45 | 3.73 | 3.47 | 3.58 | 4.74 | 5 |\n| b10 | auc5 | PRROC | 0.14 | 0.14 | 0.17 | 0.14 | 0.16 | 0.28 | 5 |\n| b10 | auc5 | ROCR | 1.57 | 1.59 | 1.69 | 1.60 | 1.63 | 2.06 | 5 |\n\n## Documentation\n\n- [Introduction to\n prcbench](https://evalclass.github.io/prcbench/articles/introduction.html)\n – a package vignette that contains the descriptions of the functions\n with several useful examples. View the vignette with\n `vignette(\"introduction\", package = \"prcbench\")` in R.\n\n- [Help pages](https://evalclass.github.io/prcbench/reference/) – all\n the functions including the S3 generics have their own help pages with\n plenty of examples. View the main help page with\n `help(package = \"prcbench\")` in R.\n\n## Installation\n\n### CRAN\n\n``` r\ninstall.packages(\"prcbench\")\n```\n\n### Dependencies\n\n`AUCCalculator` requires a Java runtime environment (\\>= 6) if\n`AUCCalculator` needs to be evaluated.\n\n### GitHub\n\nYou can install a development version of `prcbench` from [our GitHub\nrepository](https://github.com/evalclass/prcbench).\n\n``` r\ndevtools::install_github(\"evalclass/prcbench\")\n```\n\n1. Make sure you have a working development environment.\n\n - **Windows**: Install Rtools (available on the CRAN website).\n\n - **Mac**: Install Xcode from the Mac App Store.\n\n - **Linux**: Install a compiler and various development libraries\n (details vary across different flavors of Linux).\n\n2. Install `devtools` from CRAN with `install.packages(\"devtools\")`.\n\n3. Install `prcbench` from the GitHub repository with\n `devtools::install_github(\"evalclass/prcbench\")`.\n\n## Troubleshooting\n\n### microbenchmark\n\n[microbenchmark](https://cran.r-project.org/package=microbenchmark) does\nnot work on some OSs. `prcbench` uses `system.time` when\n`microbenchmark` is not available.\n\n### rJava\n\n- Some OSs require en extra configuration step after rJava installation.\n\n\n\n sudo R CMD javareconf\n\n- JDKs\n\n1. [Oracle JDK](https://www.oracle.com/java/)\n2. [OpenJDK](https://openjdk.org/)\n\n- JDKs for macOS\n\n1. [AdoptOpenJDK](https://adoptium.net/)\n2. [AdoptOpenJDK with homebrew](https://formulae.brew.sh/cask/temurin)\n\n- JRI support on macOS Big Sur – see this [Stack Overflow\n thread](https://stackoverflow.com/questions/65278552/cannot-install-rjava-on-big-sur).\n\n``` r\ninstall.packages(\"rJava\", configure.args = \"--disable-jri\")\n```\n\n## Citation\n\n*Precrec: fast and accurate precision-recall and ROC curve calculations\nin R*\n\nTakaya Saito; Marc Rehmsmeier\n\nBioinformatics 2017; 33 (1): 145-147.\n\ndoi:\n[10.1093/bioinformatics/btw570](https://doi.org/10.1093/bioinformatics/btw570)\n\n## External links\n\n- [Classifier evaluation with imbalanced\n datasets](https://classeval.wordpress.com/) – our web site that\n contains several pages with useful tips for performance evaluation on\n binary classifiers.\n\n- [The Precision-Recall Plot Is More Informative than the ROC Plot When\n Evaluating Binary Classifiers on Imbalanced\n Datasets](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432)\n – our paper that summarized potential pitfalls of ROC plots with\n imbalanced datasets and advantages of using precision-recall plots\n instead.","jobInfo":{"package":"prcbench","version":"1.1.8","parsingStatus":"success","parserVersion":2,"parsedAt":"2023-03-12T17:30:03+0000"}}

Description

Usage

Arguments

Format