shard (version 0.1.0)

shards: Shard Descriptor Creation

Description

Create shard descriptors for parallel execution with autotuning.

Produces shard descriptors (index ranges) for use with shard_map(). Supports autotuning based on worker count and memory constraints.

Usage

shards(
  n,
  block_size = "auto",
  workers = NULL,
  strategy = c("contiguous", "strided"),
  min_shards_per_worker = 4L,
  max_shards_per_worker = 64L,
  scratch_bytes_per_item = 0,
  scratch_budget = 0
)

Value

A shard_descriptor object containing:

• n: Total items

• block_size: Computed block size

• strategy: Strategy used

• shards: List of shard descriptors with id, start, end, idx fields

Arguments

n

Integer. Total number of items to shard.

block_size

Block size specification. Can be:

• "auto" (default): Autotune based on worker count

• Integer: Explicit number of items per shard

• Character: Human-readable like "1K", "10K"

workers

Integer. Number of workers for autotuning (default: pool size or detectCores - 1).

strategy

Sharding strategy: "contiguous" (default) or "strided".

min_shards_per_worker

Integer. Minimum shards per worker for load balancing (default 4).

max_shards_per_worker

Integer. Maximum shards per worker to limit overhead (default 64).

scratch_bytes_per_item

Numeric. Expected scratch memory per item for memory budgeting.

scratch_budget

Character or numeric. Total scratch memory budget (e.g., "1GB").

Examples

blocks <- shards(1e6, workers = 8)
length(blocks$shards)

blocks <- shards(1000, block_size = 100)

blocks$shards[[1]]$idx