ff (version 1.0-1)

ffm: flat-file database designed for large, multi-dimensional data arrays

Description

The function ffm and its methods allow for handling data using a flat file with memory mapped pages. It is a constructor function for ffm objects, which are multi-dimensional numeric arrays stored in a flat file.

Usage

x <- ffm(file, dim = 0, pagesize = getdefaultpagesize(), readonly = FALSE)
  ## S3 method for class 'ffm':
[(x, ...)
  ## S3 method for class 'ffm':
[(x, ...) <- value    
  ## S3 method for class 'ffm':
dim(x)
  ## S3 method for class 'ffm':
length(x)
  ## S3 method for class 'ffm':
print(x, \dots)

Arguments

file

character string giving the name of a file to load or create.

dim

dimension of the array if object should be (re-)created.

pagesize

page size or 0 for default system page size (OS-specific).

readonly

logical, specifying whether access to the resource is read-only.

value

suitable replacement value or vector of values.

x

a ffm object.

...

indices specifying elements to extract or replace (indexing); further arguments passed to or from other methods (print method).

Details

On 32-bit R platforms the indexing is limited to a maximum number of $2^{31}-1$. By using a multi-dimensional array, the data vector can be greater to overcome this limitation. The limit for ffm objects is 64 bit, even on 32-bit platforms (if supported by the virtual file system). A ffm object is stored on two files, foo.ffd for the actual binary data and foo.ffh for a header containing meta-information. As ffm objects are held by external pointers, they are copied as a reference. The connection life-time of the ffm object and its implementation part (written in C++) is under control of the garbage collector gc. To explicitly close an ffm object, one should call the garbage collector after deleting the reference(!). ff depends on the OS and file-system facilities. E.g. it is not possible to create files > 4GB on FAT32 systems. The following table gives an overview of file size limits for common file systems (see http://en.wikipedia.org/wiki/Comparison_of_file_systems for further details): ll{ File System File size limit FAT16 2GB FAT32 4GB NTFS 16GB ext2/3/4 16GB to 2TB ReiserFS 4GB (up to version 3.4) / 8TB (from version 3.5) XFS 8EB JFS 4PB HFS 2GB HFS Plus 16GB USF1 4GB to 256TB USF2 512GB to 32PB UDF 16EB }

Examples

b <- ffm("foom",c(10,20,30))   # create a multi-dimensional vector
  b[1,2,3] <- 23                 # set data cell
  b[1,2,3]                       # set data cell
  b[1:5,,] <- rnorm(20)          # complex addressing
  b[1,,]                         # show first row