Give the ensemble projection matrix and weights of the kernels in the library.
ensemble(strategy, beta_exp, error_mat, A_hat)(character) A character string indicating which ensemble strategy is to be used.
(numeric/character) A numeric value specifying the parameter
when strategy = "exp" ensemble_exp.
(matrix, n*K) A n\*K matrix indicating errors.
(list of length K) A list of projection matrices to kernel space for each kernel in the kernel library.
(matrix, n*n) The ensemble projection matrix.
(vector of length K) A vector of weights of the kernels in the library.
There are three ensemble strategies available here:
Empirical Risk Minimization (Stacking)
After obtaining the estimated errors \(\{\hat{\epsilon}_d\}_{d=1}^D\), we estimate the ensemble weights \(u=\{u_d\}_{d=1}^D\) such that it minimizes the overall error $$\hat{u}={argmin}_{u \in \Delta}\parallel \sum_{d=1}^Du_d\hat{\epsilon}_d\parallel^2 \quad where\; \Delta=\{u | u \geq 0, \parallel u \parallel_1=1\}$$ Then produce the final ensemble prediction: $$\hat{h}=\sum_{d=1}^D \hat{u}_d h_d=\sum_{d=1}^D \hat{u}_d A_{d,\hat{\lambda}_d}y=\hat{A}y$$ where \(\hat{A}=\sum_{d=1}^D \hat{u}_d A_{d,\hat{\lambda}_d}\) is the ensemble matrix.
Simple Averaging
Motivated by existing literature in omnibus kernel, we propose another way to obtain the ensemble matrix by simply choosing unsupervised weights \(u_d=1/D\) for \(d=1,2,...D\).
Exponential Weighting
Additionally, another scholar gives a new strategy to calculate weights based on the estimated errors \(\{\hat{\epsilon}_d\}_{d=1}^D\). $$u_d(\beta)=\frac{exp(-\parallel \hat{\epsilon}_d \parallel_2^2/\beta)}{\sum_{d=1}^Dexp(-\parallel \hat{\epsilon}_d \parallel_2^2/\beta)}$$
Jeremiah Zhe Liu and Brent Coull. Robust Hypothesis Test for Nonlinear Effect with Gaussian Processes. October 2017.
Xiang Zhan, Anna Plantinga, Ni Zhao, and Michael C. Wu. A fast small-sample kernel independence test for microbiome community-level association analysis. December 2017.
Arnak S. Dalalyan and Alexandre B. Tsybakov. Aggregation by Exponential Weighting and Sharp Oracle Inequalities. In Learning Theory, Lecture Notes in Computer Science, pages 97<U+2013> 111. Springer, Berlin, Heidelberg, June 2007.
mode: tuning