The EnsembleFSResult stores the results of ensemble feature selection.
It includes methods for evaluating the stability of the feature selection process and for ranking the selected features among others.
Both functions ensemble_fselect() and embedded_ensemble_fselect() return an object of this class.
as.data.table.EnsembleFSResult(x, benchmark_result = TRUE)
Returns a tabular view of the ensemble feature selection.
EnsembleFSResult -> data.table::data.table()
x (EnsembleFSResult)
benchmark_result (logical(1))
Whether to add the learner, task and resampling information from the benchmark result.
c(...)
(EnsembleFSResult, ...) -> EnsembleFSResult
Combines multiple EnsembleFSResult objects into a new EnsembleFSResult.
benchmark_result(mlr3::BenchmarkResult)
The benchmark result.
man(character(1))
Manual page for this object.
result(data.table::data.table)
Returns the result of the ensemble feature selection.
n_learners(numeric(1))
Returns the number of learners used in the ensemble feature selection.
measure(mlr3::Measure)
Returns the 'active' measure that is used in methods of this object.
active_measure(character(1))
Indicates the type of the active performance measure.
During the ensemble feature selection process, the dataset is split into multiple subsamples (train/test splits) using an initial resampling scheme. So, performance can be evaluated using one of two measures:
"outer": measure used to evaluate the performance on the test sets.
"inner": measure used for optimization and to compute performance during inner resampling on the training sets.
n_resamples(character(1))
Returns the number of times the task was initially resampled in the ensemble feature selection process.
new()Creates a new instance of this R6 class.
EnsembleFSResult$new(
result,
features,
benchmark_result = NULL,
measure,
inner_measure = NULL
)result(data.table::data.table)
The result of the ensemble feature selection.
Mandatory column names should include "resampling_iteration", "learner_id",
"features" and "n_features".
A column named as {measure$id} (scores on the test sets) must also be
always present.
The column with the performance scores on the inner resampling of the train sets is not mandatory,
but note that it should be named as {inner_measure$id}_inner to distinguish from
the {measure$id}.
features(character())
The vector of features of the task that was used in the ensemble feature
selection.
benchmark_result(mlr3::BenchmarkResult)
The benchmark result object.
measure(mlr3::Measure)
The performance measure used to evaluate the learners on the test sets generated
during the ensemble feature selection process.
By default, this serves as the 'active' measure for the methods of this object.
The active measure can be updated using the $set_active_measure() method.
inner_measure(mlr3::Measure)
The performance measure used to optimize and evaluate the learners during the inner resampling process of the training sets, generated as part of the ensemble feature selection procedure.
...(ignored).
...(ignored).
help()Opens the corresponding help page referenced by field $man.
EnsembleFSResult$help()
set_active_measure()Use this function to change the active measure.
EnsembleFSResult$set_active_measure(which = "outer")which(character(1))
Which measure from the ensemble feature selection result
to use in methods of this object.
Should be either "inner" (optimization measure used in training sets)
or "outer" (measure used in test sets, default value).
combine()Combines a second EnsembleFSResult into the current object, modifying it in-place.
If the second EnsembleFSResult (efsr) is NULL, the method returns the object unmodified.
Both objects must have the same task features and measure.
If the inner_measure differs between the objects or is NULL in either, it will be set to NULL in the combined object.
Additionally, the importance column will be removed if it is missing in either object.
If both objects contain a benchmark_result, these will be combined.
Otherwise, the combined object will have a NULL value for benchmark_result.
This method modifies the object by reference.
To preserve the original state, explicitly $clone() the object beforehand.
Alternatively, you can use the c() function, which internally calls this method.
EnsembleFSResult$combine(efsr)efsr(EnsembleFSResult)
A second EnsembleFSResult object to combine with the current object.
Returns the object itself, but modified by reference.
feature_ranking()Calculates the feature ranking via fastVoteR::rank_candidates().
EnsembleFSResult$feature_ranking(
method = "av",
use_weights = TRUE,
committee_size = NULL,
shuffle_features = TRUE
)method(character(1))
The method to calculate the feature ranking. See fastVoteR::rank_candidates()
for a complete list of available methods.
Approval voting ("av") is the default method.
use_weights(logical(1))
The default value (TRUE) uses weights equal to the performance scores
of each voter/model (or the inverse scores if the measure is minimized).
If FALSE, we treat all voters as equal and assign them all a weight equal to 1.
committee_size(integer(1))
Number of top selected features in the output ranking.
This parameter can be used to speed-up methods that build a committee sequentially
("seq_pav"), by requesting only the top N selected candidates/features
and not the complete feature ranking.
shuffle_features(logical(1))
Whether to shuffle the task features randomly before computing the ranking.
Shuffling ensures consistent random tie-breaking across methods and prevents
deterministic biases when features with equal scores are encountered.
Default is TRUE and it's advised to set a seed before running this function.
Set to FALSE if deterministic ordering of features is preferred (same as
during initialization).
The feature ranking process is built on the following framework: models act as voters, features act as candidates, and voters select certain candidates (features). The primary objective is to compile these selections into a consensus ranked list of features, effectively forming a committee.
For every feature a score is calculated, which depends on the "method" argument.
The higher the score, the higher the ranking of the feature.
Note that some methods output a feature ranking instead of a score per feature, so we always include Borda's score, which is method-agnostic, i.e. it can be used to compare the feature rankings across different methods.
We shuffle the input candidates/features so that we enforce random tie-breaking.
Users should set the same seed for consistent comparison between the different feature ranking methods and for reproducibility.
A data.table::data.table listing all the features, ordered by decreasing scores (depends on the "method"). Columns are as follows:
"feature": Feature names.
"score": Scores assigned to each feature based on the selected method (if applicable).
"norm_score": Normalized scores (if applicable), scaled to the range \([0,1]\), which can be loosely interpreted as selection probabilities (Meinshausen et al. (2010)).
"borda_score": Borda scores for method-agnostic comparison, ranging in \([0,1]\), where the top feature receives a score of 1 and the lowest-ranked feature receives a score of 0.
This column is always included so that feature ranking methods that output only rankings have also a feature-wise score.
stability()Calculates the stability of the selected features with the stabm package. The results are cached. When the same stability measure is requested again with different arguments, the cache must be reset.
EnsembleFSResult$stability(
stability_measure = "jaccard",
stability_args = NULL,
global = TRUE,
reset_cache = FALSE
)stability_measure(character(1))
The stability measure to be used.
One of the measures returned by stabm::listStabilityMeasures() in lower case.
Default is "jaccard".
stability_args(list)
Additional arguments passed to the stability measure function.
global(logical(1))
Whether to calculate the stability globally or for each learner.
reset_cache(logical(1))
If TRUE, the cached results are ignored.
A numeric() value representing the stability of the selected features.
Or a numeric() vector with the stability of the selected features for each learner.
pareto_front()This function identifies the Pareto front of the ensemble feature selection process, i.e., the set of points that represent the trade-off between the number of features and performance (e.g. classification error).
EnsembleFSResult$pareto_front(type = "empirical")type(character(1))
Specifies the type of Pareto front to return. See details.
Two options are available for the Pareto front:
"empirical" (default): returns the empirical Pareto front.
"estimated": the Pareto front points are estimated by fitting a linear model with the inversed of the number of features (\(1/x\)) as input and the associated performance scores as output.
This method is useful when the Pareto points are sparse and the front assumes a convex shape if better performance corresponds to lower measure values (e.g. classification error), or a concave shape otherwise (e.g. classification accuracy).
The estimated Pareto front will include points for a number of features ranging from 1 up to the maximum number found in the empirical Pareto front.
A data.table::data.table with columns the number of features and the performance that together form the Pareto front.
knee_points()This function implements various knee point identification (KPI) methods, which select points in the Pareto front, such that an optimal trade-off between performance and number of features is achieved. In most cases, only one such point is returned.
EnsembleFSResult$knee_points(method = "NBI", type = "empirical")method(character(1))
Type of method to use to identify the knee point. See details.
type(character(1))
Specifies the type of Pareto front to use for the identification of the knee point.
See pareto_front() method for more details.
The available KPI methods are:
"NBI" (default): The Normal-Boundary Intersection method is a geometry-based method which calculates the perpendicular distance of each point from the line connecting the first and last points of the Pareto front.
The knee point is determined as the Pareto point with the maximum distance from this line, see Das (1999).
A data.table::data.table with the knee point(s) of the Pareto front.
clone()The objects of this class are cloneable with this method.
EnsembleFSResult$clone(deep = FALSE)deepWhether to make a deep clone.
Das, I (1999). “On characterizing the 'knee' of the Pareto curve based on normal-boundary intersection.” Structural Optimization, 18(1-2), 107--115. ISSN 09344373.
Meinshausen, Nicolai, Buhlmann, Peter (2010). “Stability Selection.” Journal of the Royal Statistical Society Series B: Statistical Methodology, 72(4), 417--473. ISSN 1369-7412, tools:::Rd_expr_doi("10.1111/J.1467-9868.2010.00740.X"), 0809.2932.
# \donttest{
efsr = ensemble_fselect(
fselector = fs("rfe", n_features = 2, feature_fraction = 0.8),
task = tsk("sonar"),
learners = lrns(c("classif.rpart", "classif.featureless")),
init_resampling = rsmp("subsampling", repeats = 2),
inner_resampling = rsmp("cv", folds = 3),
inner_measure = msr("classif.ce"),
measure = msr("classif.acc"),
terminator = trm("none")
)
# contains the benchmark result
efsr$benchmark_result
# contains the selected features for each iteration
efsr$result
# returns the stability of the selected features
efsr$stability(stability_measure = "jaccard")
# returns a ranking of all features
head(efsr$feature_ranking())
# returns the empirical pareto front, i.e. n_features vs measure (error)
efsr$pareto_front()
# returns the knee points (optimal trade-off between n_features and performance)
efsr$knee_points()
# change to use the inner optimization measure
efsr$set_active_measure(which = "inner")
# Pareto front is calculated on the inner measure
efsr$pareto_front()
# }
Run the code above in your browser using DataLab