hdd (version 0.1.1)

hdd_merge: Merges data to a HDD file

Description

This function merges in-memory/HDD data to a HDD file.

Usage

hdd_merge(
  x,
  y,
  newfile,
  chunkMB,
  rowsPerChunk,
  all = FALSE,
  all.x = all,
  all.y = all,
  allow.cartesian = FALSE,
  replace = FALSE,
  verbose
)

Value

This function does not return anything. It applies the merging between two potentially large (out of memory) data set and saves them on disk at the location of newfile, the destination folder which will be populated with .fst files representing chunks of the resulting merge.

To interact with the data (on disk) newly created, use the function hdd().

Arguments

x

A HDD object or a data.frame.

y

A data set either a data.frame of a HDD object.

newfile

Destination of the result, i.e., a destination folder that will receive the HDD data.

chunkMB

Numeric, default is missing. If provided, the data 'x' is split in chunks of 'chunkMB' MB and the merge is applied chunkwise.

rowsPerChunk

Integer, default is missing. If provided, the data 'x' is split in chunks of 'rowsPerChunk' rows and the merge is applied chunkwise.

all

Default is FALSE.

all.x

Default is all.

all.y

Default is all.

allow.cartesian

Logical: whether to allow cartesian merge. Defaults to FALSE.

replace

Default is FALSE: if the destination folder already contains data, whether to replace it.

verbose

Numeric. Whether information on the advancement should be displayed. If equal to 0, nothing is displayed. By default it is equal to 1 if the size of x is greater than 1GB.

Author

Laurent Berge

Details

If x (resp y) is a HDD object, then the merging will be operated chunkwise, with the original chunks of the objects. To change the size of the chunks for x: you can use the argument chunkMB or rowsPerChunk.

To change the chunk size of y, you can rewrite y with a new chunk size using write_hdd.

Note that the merging operation could also be achieved with hdd_slice (although it would require setting up an ad hoc function).

See Also

See hdd, sub-.hdd and cash-.hdd for the extraction and manipulation of out of memory data. For importation of HDD data sets from text files: see txt2hdd.

See hdd_slice to apply functions to chunks of data (and create HDD objects) and hdd_merge to merge large files.

To create/reshape HDD objects from memory or from other HDD objects, see write_hdd.

To display general information from HDD objects: origin, summary.hdd, print.hdd, dim.hdd and names.hdd.

Examples

# Toy example with iris data

# Cartesian merge example
iris_bis = iris
names(iris_bis) = c(paste0("x_", 1:4), "species_bis")
# We must have a common key on which to merge
iris_bis$id = iris$id = 1

# merge, we chunk 'x' by 50 rows
hdd_path = tempfile()
hdd_merge(iris, iris_bis, newfile = hdd_path,
		  rowsPerChunk = 50, allow.cartesian = TRUE)

base_merged = hdd(hdd_path)
summary(base_merged)
print(base_merged)