reproducible (version 3.2.1)

prepInputs: Download and optionally post-process files

Description

This function can be used to prepare R objects from remote or local data sources. The object of this function is to provide a reproducible version of a series of commonly used steps for getting, loading, and processing data. This function has two stages: Getting data (download, extracting from archives, loading into R) and post-processing (for Spatial* and Raster* objects, this is crop, reproject, mask/intersect). To trigger the first stage, provide url or archive. To trigger the second stage, provide studyArea or rasterToMatch. See examples.

Usage

prepInputs(
  targetFile = NULL,
  url = NULL,
  archive = NULL,
  alsoExtract = NULL,
  destinationPath = getOption("reproducible.destinationPath", "."),
  fun = NULL,
  quick = getOption("reproducible.quick"),
  overwrite = getOption("reproducible.overwrite", FALSE),
  purge = FALSE,
  useCache = getOption("reproducible.useCache", 2),
  .tempPath,
  verbose = getOption("reproducible.verbose", 1),
  ...
)

Value

This is an omnibus function that will return an R object that will have resulted from the running of preProcess() and postProcess() or postProcessTo(). Thus, if it is a GIS object, it may have been cropped, reprojected, "fixed", masked, and written to disk.

Arguments

targetFile

Character string giving the filename (without relative or absolute path) to the eventual file (raster, shapefile, csv, etc.) after downloading and extracting from a zip or tar archive. This is the file before it is passed to postProcess. The internal checksumming does not checksum the file after it is postProcessed (e.g., cropped/reprojected/masked). Using Cache around prepInputs will do a sufficient job in these cases. See table in preProcess().

url

Optional character string indicating the URL to download from. If not specified, then no download will be attempted. If not entry exists in the CHECKSUMS.txt (in destinationPath), an entry will be created or appended to. This CHECKSUMS.txt entry will be used in subsequent calls to prepInputs or preProcess, comparing the file on hand with the ad hoc CHECKSUMS.txt. See table in preProcess().

archive

Optional character string giving the path of an archive containing targetFile, or a vector giving a set of nested archives (e.g., c("xxx.tar", "inner.zip", "inner.rar")). If there is/are (an) inner archive(s), but they are unknown, the function will try all until it finds the targetFile. See table in preProcess(). If it is NA, then it will not attempt to see it as an archive, even if it has archive-like file extension (e.g., .zip). This may be useful when an R function is expecting an archive directly.

alsoExtract

Which files to fetch or extract in addition to targetFile. Spatial data rarely arrives as one file -- a shapefile is five or six, and a raster often has a .aux.xml, .ovr or .tfw alongside carrying its attribute table, overviews or georeferencing -- so this controls how much of that company comes along.

The default depends on where the files are coming from, because "everything" only means something when a container bounds it:

alsoExtract	from an archive	from a url
NULL (default)	every file in the archive	as for "similar"
"similar"	files sharing targetFile's name without its extension	same, discovered by listing the remote directory
NA or "none"	targetFile only	targetFile only
a character vector	those files only	those files only

An archive bounds what "all files" can mean, so NULL extracts all of it. A url has no such bound -- the server may hold thousands of unrelated files -- so NULL there means targetFile and its companions, never the whole remote directory.

Each element of a character vector may also be a regular expression: if an element does not match any archive member literally (by relative path or basename), it is passed to grep() against the archive's file list and all matching members are extracted. For example, alsoExtract = "CMD_sm|CMD_sp" extracts every file whose name contains CMD_sm or CMD_sp. See table in preProcess().

destinationPath

Character string of a directory in which to download and save the file that comes from url and is also where the function will look for archive or targetFile. To prevent repeated downloads in different locations, the user can also set options("reproducible.destinationPathShared") to one or more local file paths to search for the file before attempting to download. Default for that option is NULL meaning do not search locally. The previous name options("reproducible.inputPaths") is still accepted as a backwards-compatible alias.

fun

Optional. If specified, this will attempt to load whatever file was downloaded during preProcess via dlFun. This can be either a function (e.g., sf::st_read), character string (e.g., "base::load"), NA (for no loading, useful if dlFun already loaded the file) or if extra arguments are required in the function call, it must be a call naming targetFile (e.g., sf::st_read(targetFile, quiet = TRUE)) as the file path to the file to load. See details and examples below.

quick

Logical. This is passed internally to Checksums() (the quickCheck argument), and to Cache() (the quick argument). This results in faster, though less robust checking of inputs. See the respective functions.

overwrite

Logical. Passed to writeTo (possibly inside postProcess) and postProcess.

purge

Logical or Integer. 0/FALSE (default) keeps existing CHECKSUMS.txt file and prepInputs will write or append to it. 1/TRUE will deleted the entire CHECKSUMS.txt file. Other options, see details.

useCache

Passed to Cache in various places. Defaults to getOption("reproducible.useCache", 2L) in prepInputs, and getOption("reproducible.useCache", FALSE) if calling any of the inner functions manually. For prepInputs, this mean it will use Cache only up to 2 nested levels, which includes preProcess. postProcess and its nested *Input functions (e.g., cropInputs, projectInputs, maskInputs) are no longer internally cached, as terra processing speeds mean internal caching is more time consuming. We recommend caching the full prepInputs call instead (e.g. prepInputs(...) |> Cache()).

.tempPath

Optional temporary path for internal file intermediate steps. Will be cleared on.exit from this function.

verbose

Numeric, -1 silent (where possible), 0 being very quiet, 1 showing more messaging, 2 being more messaging, etc. Default is 1. Above 3 will output much more information about the internals of Caching, which may help diagnose Caching challenges. Can set globally with an option, e.g., options('reproducible.verbose' = 0) to reduce to minimal

...

Additional arguments passed to postProcess() and Cache(). Since ... is passed to postProcess(), these will ... will also be passed into the inner functions, e.g., cropInputs(). Possibly useful other arguments include dlFun which is passed to preProcess. See details and examples.

Stage 1 - Getting data

See preProcess() for combinations of arguments.

1. Download from the web via either googledrive::drive_download(), utils::download.file();

2. Extract from archive using unzip() or untar();

3. Load into R using terra::rast, sf::st_read, or any other function passed in with fun;

4. Checksumming of all files during this process. This is put into a CHECKSUMS.txt file in the destinationPath, appending if it is already there, overwriting the entries for same files if entries already exist.

Stage 2 - Post processing

This will be triggered if either rasterToMatch or studyArea is supplied.

1. Fix errors. Currently only errors fixed are for SpatialPolygons using buffer(..., width = 0);

2. Crop using cropTo();

3. Project using projectTo();

4. Mask using maskTo();

5. write the file to disk via writeTo().

NOTE: checksumming does not occur during the post-processing stage, as there are no file downloads. To achieve fast results, wrap prepInputs with Cache.

postProcessing of Spat*, sf, Raster* and Spatial* objects:

The following has been DEPRECATED because there are a sufficient number of ambiguities that this has been changed in favour of from and the *to family. See postProcessTo().

DEPRECATED: If rasterToMatch or studyArea are used, then this will trigger several subsequent functions, specifically the sequence, Crop, reproject, mask, which appears to be a common sequence while preparing spatial data from diverse sources. See postProcess() documentation section on Backwards compatibility with rasterToMatch and/or studyArea arguments to understand various combinations of rasterToMatch and/or studyArea.

<code>fun</code>

fun offers the ability to pass any custom function with which to load the file obtained by preProcess into the session. There are two cases that are dealt with: when the preProcess downloads a file (including via dlFun), fun must deal with a file; and, when preProcess creates an R object (e.g., raster::getData returns an object), fun must deal with an object.

fun can be supplied in three ways: a function, a character string (i.e., a function name as a string), or an expression. If a character string or function, is should have the package name e.g., "terra::rast" or as an actual function, e.g., base::readRDS. In these cases, it will evaluate this function call while passing targetFile as the first argument. These will only work in the simplest of cases.

When more precision is required, the full call can be written and where the filename can be referred to as targetFile if the function is loading a file. If preProcess returns an object, fun should be set to fun = NA.

If there is a custom function call, is not in a package, prepInputs may not find it. In such cases, simply pass the function as a named argument (with same name as function) to prepInputs. See examples. NOTE: passing fun = NA will skip loading object into R. Note this will essentially replicate the functionality of simply calling preProcess directly.

<code>purge</code>

In options for control of purging the CHECKSUMS.txt file are:

0

keep file

1

delete file in destinationPath, all records of downloads need to be rebuilt

2

delete entry with same targetFile

4

delete entry with same alsoExtract

3

delete entry with same archive

5

delete entry with same targetFile & alsoExtract

6

delete entry with same targetFile, alsoExtract & archive

7

delete entry that same targetFile, alsoExtract & archive & url

will only remove entries in the CHECKSUMS.txt that are associated with targetFile, alsoExtract or archive When prepInputs is called, it will write or append to a (if already exists) CHECKSUMS.txt file. If the CHECKSUMS.txt is not correct, use this argument to remove it.

Author

Eliot McIntire, Jean Marchal, and Tati Micheletti

See Also

postProcessTo(), downloadFile(), extractFromArchive(), postProcess().

Examples

if (requireNamespace("terra", quietly = TRUE) &&
  requireNamespace("withr", quietly = TRUE)) {
  library(reproducible)
  withr::local_dir(withr::local_tempdir())
  # Make a dummy study area map -- user would supply this normally
  coords <- structure(c(-122.9, -116.1, -99.2, -106, -122.9, 59.9, 65.7, 63.6, 54.8, 59.9),
    dim = c(5L, 2L)
  )
  studyArea <- terra::vect(coords, "polygons")
  terra::crs(studyArea) <- "+proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0"
  # Make dummy "large" map that must be cropped to the study area
  outerSA <- terra::buffer(studyArea, 50000)
  terra::crs(outerSA) <- "+proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0"
  tf <- normPath(file.path(tempdir2(), "prepInputs2.shp"))
  terra::writeVector(outerSA, tf)

  # run prepInputs -- load file, postProcess it to the studyArea

  studyArea2 <- prepInputs(
    targetFile = tf, to = studyArea,
    fun = "terra::vect",
    destinationPath = tempdir2()
  ) |>
    suppressWarnings() # not relevant warning here

  # clean up
  unlink("CHECKSUMS.txt")

  ##########################################
  # Remote file using `url`
  ##########################################
  if (identical(Sys.getenv("NOT_CRAN"), "true")) {
    data.table::setDTthreads(2)
    # download a zip file from internet, unzip all files, load as shapefile
    # Don't know all the files -- prepInputs will guess: if the downloaded file is an
    #   archive, extract all files, then if there is a .shp, load it with sf::st_read
    dPath <- file.path(tempdir2(), "ecozones")
    shpUrl <- paste0(
      "https://github.com/PredictiveEcology/reproducible/releases/download/v3.1.1/",
      "ecozone_shp.zip"
    )

    # Wrapped in a try because this needs internet
    shpEcozone <- try(prepInputs(destinationPath = dPath, url = shpUrl))

    ##########################################
    # Local archive -- the same file, shipped with the package
    ##########################################
    # The remaining examples use the copy in `inst/ex`, so they need no internet
    dPath <- file.path(tempdir2(), "ecozonesLocal")
    checkPath(dPath, create = TRUE)
    file.copy(system.file("ex/ecozone_shp.zip", package = "reproducible"), dPath)

    shpEcozone <- prepInputs(destinationPath = dPath, archive = "ecozone_shp.zip")

    # Robust to partial file deletions: the missing files are re-extracted
    unlink(dir(file.path(dPath, "Ecozones"), full.names = TRUE)[1:2])
    shpEcozone <- prepInputs(destinationPath = dPath, archive = "ecozone_shp.zip")

    # Once this is done, can be more precise in operational code:
    #  specify targetFile, alsoExtract, and fun, wrap with Cache
    ecozoneFilename <- file.path(dPath, "Ecozones", "ecozones.shp")
    # Note, you don't need to "alsoExtract" the archive... if the archive is not there, but the
    #   targetFile is there, it will not re-extract the archive.
    ecozoneFiles <- c("ecozones.dbf", "ecozones.prj", "ecozones.shp", "ecozones.shx")
    shpEcozone <- prepInputs(
      targetFile = ecozoneFilename,
      archive = "ecozone_shp.zip",
      fun = "terra::vect",
      alsoExtract = ecozoneFiles,
      destinationPath = dPath
    )

    # Add a study area to Crop and Mask to
    # Create a "study area"
    coords <- structure(c(-122.98, -116.1, -99.2, -106, -122.98, 59.9, 65.73, 63.58, 54.79, 59.9),
      dim = c(5L, 2L)
    )
    studyArea <- terra::vect(coords, "polygons")
    terra::crs(studyArea) <- "+proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0"

    shpEcozoneSm <- Cache(prepInputs,
      archive = "ecozone_shp.zip",
      targetFile = reproducible::asPath(ecozoneFilename),
      alsoExtract = reproducible::asPath(ecozoneFiles),
      studyArea = studyArea,
      fun = "terra::vect",
      destinationPath = dPath,
      writeTo = "EcozoneFile.shp"
    ) # passed to determineFilename

    terra::plot(shpEcozone[, 1])
    terra::plot(shpEcozoneSm[, 1], add = TRUE, col = "red")
  }
  withr::deferred_run()
}

## Using quoted dlFun and fun -- this is not intended to be run but used as a template
## prepInputs(..., fun = customFun(x = targetFile), customFun = customFun)
##   # or more complex
##  test5 <- prepInputs(
##   targetFile = targetFileLuxRDS,
##   dlFun =
##     getDataFn(name = "GADM", country = "LUX", level = 0) # preProcess keeps file from this!
##   ,
##   fun = {
##     out <- readRDS(targetFile)
##     sf::st_as_sf(out)}
##  )