Learn R Programming

performanceEstimation (version 1.1.0)

An Infra-Structure for Performance Estimation of Predictive Models

Description

An infra-structure for estimating the predictive performance of predictive models. In this context, it can also be used to compare and/or select among different alternative ways of solving one or more predictive tasks. The main goal of the package is to provide a generic infra-structure to estimate the values of different metrics of predictive performance using different estimation procedures. These estimation tasks can be applied to any solutions (workflows) to the predictive tasks. The package provides easy to use standard workflows that allow the usage of any available R modeling algorithm together with some pre-defined data pre-processing steps and also prediction post- processing methods. It also provides means for addressing issues related with the statistical significance of the observed differences.

Copy Link

Version

Install

install.packages('performanceEstimation')

Monthly Downloads

742

Version

1.1.0

License

GPL (>= 2)

Issues

Pull Requests

Stars

Forks

Repository

https://github.com/ltorgo/performanceEstimation

Maintainer

Luis Torgo

Last Published

October 13th, 2016

Functions in performanceEstimation (1.1.0)

Performance estimation using (e0 or .632) bootstrap

Bootstrap-class

Class "Bootstrap"

EstimationMethod-class

Class "EstimationMethod"

EstCommon-class

Class "EstCommon"

CD diagrams for the post-hoc Boferroni-Dunn test

classificationMetrics

Calculate some standard classification evaluation metrics of predictive performance

ComparisonResults-class

Class "ComparisonResults"

CDdiagram.Nemenyi

CD diagrams for the post-hoc Nemenyi test

Performance estimation using cross validation

estimationSummary

Obtain a set of descriptive statistics of the scores of a workflow on a task

EstimationResults-class

Class "EstimationResults"

is.classification

Check if a certain predictive task is a classification problem

getIterationsPreds

Obtaining the predictions returned by a workflow when applied to a task, on a particular iteration of the estimation process, or on all iterations

Obtaining the metric scores on the different iterations for a workflow / task combination

Class "Holdout"

EstimationTask-class

Class "EstimationTask"

getIterationsInfo

Obtaining the information returned by a workflow when applied to a task, on a particular iteration of the estimation process or on all iterations

Performance estimation using holdout and random resampling

Obtain the workflow object corresponding to an ID

Performance estimation using Leave One Out Cross Validation

MonteCarlo-class

Class "MonteCarlo"

Performance estimation for time series prediction tasks using Monte Carlo

Obtains a summary of the individual metric scores obtained by each workflow on a set of tasks.

pairedComparisons

Statistical hypothesis testing on the observed paired differences in estimated performance.

Check if a certain predictive task is a regression problem

mergeEstimationRes

Merging several ComparisonResults class objects

The evaluation metrics estimated in an experiment

Fill in NA values with the values of the nearest neighbours

Obtain the target variable values of a prediction task

Provide a ranking of workflows involved in an estimation process.

SMOTE algorithm for unbalanced classification problems

A function for applying post-processing steps to the predictions of a model

regressionMetrics

Calculate some standard regression evaluation metrics of predictive performance

Run a workflow on a predictive task

performanceEstimation

Estimate the predictive performance of modeling alternatives on different predictive tasks

Class "PredTask"

Obtains a list with the set of paired differences that are statistically significant according to a p-value threshold

Obtains a dplyr data frame table object containing all the results of an experiment

Methods for Function subset in Package performanceEstimation

The prediction tasks involved in an estimation experiment

A function for applying data pre-processing steps

A function implementing a standard workflow for prediction tasks

workflowVariants

Generate (parameter) variants of a workflow

The IDs of the workflows involved in an estimation experiment

Obtain the workflow that best performed in terms of a metric on a task

A function implementing sliding and growing window standard workflows for time series forecasting tasks

Class "Workflow"

Obtain the best scores from a performance estimation experiment