Learn R Programming

⚠️There's a newer version (1.1.0) of this package.Take me there.

performanceEstimation (version 1.0.2)

An Infra-Structure for Performance Estimation of Predictive Models

Description

An infra-structure for estimating the predictive performance of predictive models. In this context, it can also be used to compare and/or select among different alternative ways of solving one or more predictive tasks. The main goal of the package is to provide a generic infra-structure to estimate the values of different metrics of predictive performance using different estimation procedures. These estimation tasks can be applied to any solutions (workflows) to the predictive tasks. The package provides easy to use standard workflows that allow the usage of any available R modeling algorithm together with some pre-defined data pre-processing steps and also prediction post-processing methods. It also provides means for addressing issues related with the statistical significance of the observed differences.

Copy Link

Version

Install

install.packages('performanceEstimation')

Monthly Downloads

52

Version

1.0.2

License

GPL (>= 2)

Issues

Pull Requests

Stars

Forks

Repository

https://github.com/ltorgo/performanceEstimation

Maintainer

Luis Torgo

Last Published

September 7th, 2015

Functions in performanceEstimation (1.0.2)

Performance estimation using (e0 or .632) bootstrap

CD diagrams for the post-hoc Boferroni-Dunn test

EstimationMethod-class

Class "EstimationMethod"

A function for applying post-processing steps to the predictions of a model

Bootstrap-class

Class "Bootstrap"

Performance estimation using cross validation

pairedComparisons

Statistical hypothesis testing on the observed paired differences in estimated performance.

Obtains a list with the set of paired differences that are statistically significant according to a p-value threshold

regressionMetrics

Calculate some standard regression evaluation metrics of predictive performance

Provide a ranking of workflows involved in an estimation process.

Methods for Function subset in Package performanceEstimation

is.classification

Check if a certain predictive task is a classification problem

EstimationTask-class

Class "EstimationTask"

Run a workflow on a predictive task

Obtain the workflow that best performed in terms of a metric on a task

ComparisonResults-class

Class "ComparisonResults"

Obtain the workflow object corresponding to an ID

EstCommon-class

Class "EstCommon"

A function implementing a standard workflow for prediction tasks

Class "PredTask"

estimationSummary

Obtain a set of descriptive statistics of the scores of a workflow on a task

Performance estimation using holdout and random resampling

Obtains a summary of the individual metric scores obtained by each workflow on a set of tasks.

The evaluation metrics estimated in an experiment

Obtain the target variable values of a prediction task

A function implementing sliding and growing window standard workflows for time series forecasting tasks

performanceEstimation

Estimate the predictive performance of modeling alternatives on different predictive tasks

Obtain the best scores from a performance estimation experiment

mergeEstimationRes

Merging several ComparisonResults class objects

CDdiagram.Nemenyi

CD diagrams for the post-hoc Nemenyi test

getIterationsInfo

Obtaining the information returned by a workflow when applied to a task, on a particular iteration of the estimation process or on all iterations

workflowVariants

Generate (parameter) variants of a workflow

Obtaining the metric scores on the different iterations for a workflow / task combination

The IDs of the workflows involved in an estimation experiment

EstimationResults-class

Class "EstimationResults"

MonteCarlo-class

Class "MonteCarlo"

A function for applying data pre-processing steps

Performance estimation using Leave One Out Cross Validation

getIterationsPreds

Obtaining the predictions returned by a workflow when applied to a task, on a particular iteration of the estimation process, or on all iterations

Fill in NA values with the values of the nearest neighbours

The prediction tasks involved in an estimation experiment

Check if a certain predictive task is a regression problem

SMOTE algorithm for unbalanced classification problems

Class "Holdout"

Class "Workflow"

classificationMetrics

Calculate some standard classification evaluation metrics of predictive performance

Performance estimation for time series prediction tasks using Monte Carlo