This function gives the penalty functions
penalty(x, n, p, q, beta, lam1, pmethod, p1, lam2)
the penalty function.
matrix of covariates.
the sample size.
the number of predictors.
the number of environment factors.
the coefficient vector.
the tuning parameter lambda1 for individual penalty.
the penalization method. "mixed" refers to MCP penalty to individual main effects and group MCP penalty to interactions; "individual" means MCP penalty to all effects.
the number of gene factors.
the tuning parameter lambda2 for group penalty.