arkdb (version 0.0.18)

process_chunks: process a table in chunks

Description

process a table in chunks

Usage

process_chunks(
  file,
  process_fn,
  streamable_table = NULL,
  lines = 50000L,
  encoding = Sys.getenv("encoding", "UTF-8"),
  ...
)

Arguments

file

path to a file

process_fn

a function of a chunk

streamable_table

interface for serializing/deserializing in chunks

lines

number of lines to read in a chunk.

encoding

encoding to be assumed for input files.

...

additional arguments to streamable_table$read method.

Examples

con <- system.file("extdata/mtcars.tsv.gz", package = "arkdb")
dummy <- function(x) message(paste(dim(x), collapse = " x "))
process_chunks(con, dummy, lines = 8)