reproducible (version 3.2.1)

makeUrlRemap: Build a URL remap function from a manifest

Description

Convenience constructor for the reproducible.urlRemap option (see preProcess()). Given a data.frame with (at least) columns filename and url, it returns a function function(url, filename) suitable for options(reproducible.urlRemap = ...). The returned function matches on the basename of the resolved filename: when a manifest row's filename matches, its url is returned, so the download is redirected there (and, if that URL supports HTTP Range requests, the parallel download path applies). When there is no match it returns NULL, so the original URL is kept.

Usage

makeUrlRemap(manifest)

Value

A function of (url, filename) returning a replacement URL, or NULL

to keep the original.

Arguments

manifest

A data.frame (or data.table) with at least the character columns filename and url. filename is matched against the basename of the file being downloaded. An optional id column (Google Drive file id) is matched secondarily, by the id parsed from the Drive url, when no filename is available (e.g. an unauthenticated Drive download). An optional type column ("file"/"dir") marks directory-remap rows (a Drive folder id -> bucket prefix-listing URL), used to enumerate a Drive folder from the public bucket without authentication.

Details

An optional id column (also accepted as googledriveId, googledrive_id, driveId or gid) holding the Google Drive file id enables a secondary match by id: when the resolved filename is unavailable — e.g. an unauthenticated session that cannot read a Drive file's metadata — the id is parsed from the Drive url and matched against this column. This lets a download be redirected to the (public) mirror with no authentication at all.

An optional type column ("file"/"dir") enables directory remaps: a "dir" row maps a Google Drive folder id (id column) to a bucket prefix-listing URL (its url). When preProcess() downloads such a folder, it enumerates the folder's files from that public listing instead of googledrive::drive_ls(), so listing the folder needs no authentication. Rows without a type column (or with type = "file") are ordinary file remaps, unchanged. buckethost::makeMirrorManifest(directories = TRUE) emits such a manifest.

The manifest itself — and the responsibility for keeping it current — lives with the user (for example, a community-maintained mirror manifest); reproducible hard-codes no mirror URLs.

Calling makeUrlRemap() yourself is optional: you can also assign the manifest data.frame (or a CSV path/URL) directly to the option, e.g. options(reproducible.urlRemap = read.csv("manifest.csv")), and reproducible will build (and cache) the remap function internally. See the urlRemap entry in reproducibleOptions().

See Also

preProcess() for the reproducible.urlRemap option.

Examples

manifest <- data.frame(
  filename = "SCANFI_att_biomass_2010_v2_20260119.tif",
  url = paste0(
    "https://object-arbutus.cloud.computecanada.ca/predictiveecology/",
    "SCANFI_v2/2010/SCANFI_att_biomass_2010_v2_20260119.tif"
  )
)
remap <- makeUrlRemap(manifest)

# matched by basename, whatever the original url was
remap("https://drive.google.com/file/d/abc123/view",
      "SCANFI_att_biomass_2010_v2_20260119.tif")
remap("https://example.com/other.tif", "other.tif") # NULL -- keep original

# to apply it to every download in a session:
#   options(reproducible.urlRemap = remap)