BayesSummaryStatLM (version 2.0)

read.regress.data.ff: Read in Tabulated Data and Compute Summary Statistics

Description

This function reads in tabulated data sets and produces summary statistics needed for Bayesian linear regression models for use in the function bayes.regress() (in this package). Big data sets that are too large to fit into R memory are handled using functions from package ff. The function takes as input data files with the predictor variables \(X\) and response values \(Y\), and returns the summary statistics \(X'X\), \(X'Y\) and \(Y'Y\) that are used as an input to the function bayes.regress() (in this package) for Bayesian linear regression models. The function supports reading data sets that are split across multiple files.

Usage

read.regress.data.ff(filename=NULL,predictor.cols=NA,response.col=NA,update.summaries=NULL
                   , fileEncoding = "", nrows = -1, first.rows = 1e5, next.rows = 1e5
                   , levels = NULL, appendLevels = TRUE,FUN = "read.table",transFUN = NULL
                   , asffdf_args = list(), BATCHBYTES = getOption("ffbatchbytes")
                   , VERBOSE = FALSE, header = FALSE, sep = ",", quote = "\"'", dec = "."
                   , numerals = c("allow.loss", "warn.loss", "no.loss")
                   , na.strings = "NA", colClasses = "numeric", skip = 0
                   , check.names = TRUE, fill = TRUE, strip.white = FALSE
                   , blank.lines.skip = TRUE, comment.char = "#", allowEscapes = FALSE
                   , flush = FALSE, skipNul = FALSE)

Arguments

filename

the name of a file or a list of file names, from which the data will be read. By default it is assumed that the file(s) contain data in comma separated format; this can be changed using the sep argument. File names must be in the format acceptable by standard functions such as read.table. Default = NULL.

predictor.cols

a vector of integers that specifies the columns to treat as predictor variables, to create the design matrix \(X\). By default, all columns after the first are treated as columns with predictor variables. Default = NA.

response.col

an integer that specifies the column that contains the response variable values, to create the response vector \(Y\). By default, the first column that is not used as a predictor variable column is selected as the response values column. Default = NA.

update.summaries

The name of the R object containing previously-calculated summary statistics (if applicable), to be updated with new data. This must be a list similar in structure to the returned value, containing entries xtx, xty, yty and numsamp.data of appropriate dimensions and class that match the data contained in filename. Default = NULL.

first.rows

the number of rows to read in the first chunk of data. Default = 100,000.

next.rows

the number of rows to read in the remaining chunks of data. Default = 100,000.

sep

the character that separates the columns of data. Default = ",".

fileEncoding

a string that describes the file's character encoding

nrows

an integer specifying how many rows should be read from the file

levels

an optional list of items with col.names or factor columns. See read.table.ffdf().

appendLevels

a logical vector of permission to expand levels

FUN

specifies which standard R function is used to read the data.

transFUN

an optional filtering function to be applied to each chunk of data. See read.table.ffdf().

asffdf_args

an optional list of parameters to be passed to as.ffdf() function.

BATCHBYTES

an integer limiting the size of the data.frame used to store each chunk of the data

VERBOSE

See read.table.ffdf().

header

a logical value indicating if the first row is the header row

quote

a character string specifying which character will be treated as quoting characters

dec

a character used for decimal dot

numerals

see read.table().

na.strings

strings treated as NA values

colClasses

a vector that describes the data types in each column. Numeric by default.

skip

how many first lines in the file should be skipped

check.names

see read.table().

fill

logical value that turns on automatic padding of the rows in case they have different lengths

strip.white

affects the processing of the columns with declared character type. See scan().

blank.lines.skip

logical value, indicating whether empty lines should be ignored

comment.char

character specifying the comment marker

allowEscapes

logical. If TRUE, the escaped strings will be parsed. See scan().

flush

see read.table().

skipNul

logical: should nuls be skipped?

Value

The returned value for the read.regress.data.ff() function is a list containing the summary statistics named xtx (for \(X'X\)), xty (for \(X'Y\)), yty (for \(Y'Y\)) and the total number of data values numsamp.data. The summary statistic xtx contains a square matrix obtained by computing a dot product of the predictor variables data \(X\) with itself; a leading column of 1's is added to \(X\) for the y-intercept term. xty contains the vector obtained by computing the dot product of the transposed predictor variables data \(X\) with response variable data \(Y\); a leading column of 1's is added to \(X\) for the y-intercept term. yty contains the dot product of the response variable data \(Y\) with itself. numsamp.data is the number of data values read from the data file(s); this number may be smaller than the number of rows in the data file, since some of the rows with missing data may be skipped according to specified function arguments. The summary statistics \(X'X\), \(X'Y\) and \(Y'Y\) are summed over data chunks by the following, for \(m = 1,...,M\) chunks: $$X'X = sum_{m=1}^{M} (X'_{m})( X_{m})$$ $$X'Y = sum_{m=1}^{M} (X'_{m})( Y_{m})$$ $$Y'Y = sum_{m=1}^{M} (Y'_{m})( Y_{m})$$

The returned values are used as input to the function bayes.regress() (in this package). Note that the matrix X is given a leading column of 1's by default, for the y-intercept term of the Bayesian linear regression model. This can be removed by specifying a model with zero intercept in the function bayes.regress() (see bayes.regress() help for details).

Details

The function reads in data and computes summary statistics to be used in Bayesian linear regression by the function bayes.regress() (in this package). The function assumes the linear regression model will have a non-zero y-intercept; this option can be changed in the bayes.regress() function (see bayes.regress() help for details).

References

Carlin, B.P. and Louis, T.A. (2009) Bayesian Methods for Data Analysis, 3rd ed., Boca Raton, FL: Chapman and Hall/CRC Press.

Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A. and Rubin, D.B. (2013) Bayesian Data Analysis, 3rd ed., Boca Raton, FL: Chapman and Hall/CRC Press.

Adler, D., Glaser, C., Nenadic, O., Oehlschlagel, J. and Zucchini, W. (2013) ff: memory-efficient storage of large data on disk and fast access functions. R package: https://CRAN.R-project.org/package=ff.

Examples

# NOT RUN {
# The package includes several example data files, illustrated here.

#########
# Example
#########
# The following command finds the location of the data file
# that includes 4 predictor variables and 20,000 simulated data values.

filename <- system.file('data/regressiondata.nz.all.csv.gz', package='BayesSummaryStatLM')

# The file is formatted so that the simulated response variable is in the
# first column, and columns 2 to 5 contain simulated predictor variables.
# The simulated coefficients are: beta <- c(0.76, -0.92, 0.64, 0.57, -1.65), 
# where the first value is the y-intercept term in the Bayesian linear 
# regression model.  The sigma-squared term, i.e. the variance of the normally 
# distributed error terms, is simulated as: sigmasq <- 0.25

## Next, read the data and compute the summary statistics using the
# "read.regress.data.ff()" function.  By default, the first column is assumed 
# to be the response variable, and the remaining columns are assumed to contain
# predictor variable values.  The function will check if the file exists and 
# can be read.

data.values <- read.regress.data.ff(filename)
data.values

# }