vald.extractor (version 0.1.1)

fetch_vald_batch: Robust Batch Extraction of VALD Trials

Description

Implements chunked trial extraction from VALD ForceDecks API with fault-tolerant error handling. This function prevents timeout errors and memory issues when working with large datasets by processing data in manageable chunks.

Usage

fetch_vald_batch(start_date, chunk_size = 100, verbose = TRUE)

Value

A list containing two data frames:

tests

Data frame of all tests metadata

trials

Data frame of all trials (individual repetitions) data

Arguments

start_date

Character string in ISO 8601 format (e.g., "2020-01-01T00:00:00Z"). The starting date for data extraction.

chunk_size

Integer. Number of tests to process per batch. Default is 100. Reduce this value if you experience timeout errors.

verbose

Logical. If TRUE, prints progress messages. Default is TRUE.

Details

Fetch VALD ForceDecks Data in Batches

This function first retrieves all test metadata, then iterates through tests in chunks to fetch associated trial data. Each chunk is wrapped in a tryCatch block to ensure that errors in one chunk do not halt the entire extraction process.

The chunking strategy is essential for large organizations with thousands of tests, as it prevents API timeout errors and reduces memory pressure.

Examples

# \donttest{
if (FALSE) {
  # Set VALD credentials first
  valdr::set_credentials(
    client_id = "your_client_id",
    client_secret = "your_client_secret",
    tenant_id = "your_tenant_id",
    region = "aue"
  )

  # Fetch data from 2020 onwards in chunks of 100
  vald_data <- fetch_vald_batch(
    start_date = "2020-01-01T00:00:00Z",
    chunk_size = 100
  )

  # Access tests and trials
  tests_df <- vald_data$tests
  trials_df <- vald_data$trials
}
# }