NeEDS4BigData (version 1.0.1)

modelRobustPoiSub: Model robust optimal subsampling for A- and L- optimality criteria under Poisson regression

Description

Using this function sample from big data under Poisson regression when there are more than one model to describe the data. Subsampling probabilities are obtained based on the A- and L- optimality criteria.

Usage

modelRobustPoiSub(r0,rf,Y,X,N,Apriori_probs,All_Combinations,All_Covariates)

Value

The output of modelRobustLinSub gives a list of

Beta_Data estimated model parameters for each model in a list after subsampling

Sample_L-optimality list of indexes for the initial and optimal samples obtained based on L-optimality criteria

Sample_L-optimality_MR list of indexes for the initial and model robust optimal samples obtained based on L-optimality criteria

Sample_A-optimality list of indexes for the initial and optimal samples obtained based on A-optimality criteria

Sample_A-optimality_MR list of indexes for the initial and model robust optimal samples obtained based on A-optimality criteria

Subsampling_Probability matrix of calculated subsampling probabilities for A- and L- optimality criteria

Arguments

r0

sample size for initial random sample

rf

final sample size including initial(r0) and optimal(r) samples

Y

response data or Y

X

covariate data or X matrix that has all the covariates (first column is for the intercept)

N

size of the big data

Apriori_probs

vector of a priori model probabilities that are used to obtain the model robust subsampling probabilities

All_Combinations

list of possible models that can describe the data

All_Covariates

all the covariates in the models

Details

Two stage subsampling algorithm for big data under Poisson regression for multiple models that can describe the big data.

First stage is to obtain a random sample of size \(r_0\) and estimate the model parameters for all models. Using the estimated parameters subsampling probabilities are evaluated for A-, L-optimality criteria and model averaging A-, L-optimality subsampling methods.

Through the estimated subsampling probabilities a sample of size \(r \ge r_0\) is obtained. Finally, the two samples are combined and the model parameters are estimated for all the models.

NOTE : If input parameters are not in given domain conditions necessary error messages will be provided to go further.

If \(r \ge r_0\) is not satisfied then an error message will be produced.

If the big data \(X,Y\) has any missing values then an error message will be produced.

The big data size \(N\) is compared with the sizes of \(X,Y\) and if they are not aligned an error message will be produced.

If \(0 < \alpha_{q} < 1\) for the a priori model probabilities are not satisfied an error message will be produced, where \(q=1,\ldots,Q\) and \(Q\) is the number of models in the model set.

References

mahendran2023modelNeEDS4BigData

Examples

indexes<-1:ceiling(nrow(One_million_songs)*0.5)
Original_Data<-One_million_songs[indexes,]
colnames(Original_Data)<-c("Y",paste0("X",1:ncol(Original_Data[,-1])))

# Scaling the covariate data
for (j in 2:4) {
  Original_Data[,j]<-scale(Original_Data[,j])
}

No_of_Variables<-ncol(Original_Data[,-1])
Squared_Terms<-paste0("X",1:No_of_Variables,"^2")
term_no <- 2
All_Models <- list(paste0("X",1:No_of_Variables))

Original_Data<-cbind(Original_Data,Original_Data[,-1]^2)
colnames(Original_Data)<-c("Y",paste0("X",1:No_of_Variables),
                            paste0("X",1:No_of_Variables,"^2"))

for (i in 1:No_of_Variables)
{
  x <- as.vector(combn(Squared_Terms,i,simplify = FALSE))
  for(j in 1:length(x))
  {
    All_Models[[term_no]] <- c(paste0("X",1:No_of_Variables),x[[j]])
    term_no <- term_no+1
  }
}
All_Models<-All_Models[1:4]
names(All_Models)<-paste0("Model_",1:length(All_Models))

r0<-300; rf<-rep(100*c(6,9),25);

modelRobustPoiSub(r0 = r0, rf = rf, Y = as.matrix(Original_Data[,1]),
                  X = as.matrix(Original_Data[,-1]),N = nrow(Original_Data),
                  Apriori_probs = rep(1/length(All_Models),length(All_Models)),
                  All_Combinations = All_Models,
                  All_Covariates = colnames(Original_Data)[-1])->Results

Beta_Plots<-plot_Beta(Results)