chunk_process: Process Large Dataset in Chunks
Description
Generic function to process a large dataset in manageable chunks to reduce memory usage.
Usage
chunk_process(data, chunk_size = 10000, fn, combine_fn = rbind, ...)
Value
Combined results after processing all chunks
Arguments
- data
Large data frame to process
- chunk_size
Number of rows per chunk
- fn
Function to apply to each chunk
- combine_fn
Function to combine results from chunks
- ...
Additional arguments passed to fn