shard (version 0.1.0)

stream_map: Stream over row-groups/datasets and map

Description

Applies f() to each partition and returns the list of per-partition results. This is still much cheaper than collecting the full dataset when f() returns a small summary per partition.

Usage

stream_map(x, f, ...)
# S3 method for shard_row_groups
stream_map(x, f, ...)
# S3 method for shard_dataset
stream_map(x, f, ...)

Value

A list of per-partition values, one element per row-group file.

Arguments

x

A shard_row_groups or shard_dataset handle.

f

Function (chunk, ...) -> value.

...

Passed to f().

Examples

# \donttest{
s <- schema(x = float64())
sink <- table_sink(s, mode = "row_groups")
table_write(sink, 1L, data.frame(x = rnorm(5)))
rg <- table_finalize(sink)
nrows <- stream_map(rg, nrow)
# }