Learn R Programming

reproducible (version 0.1.4)

Cache: Cache method that accommodates environments, S4 methods, Rasters

Description

Cache method that accommodates environments, S4 methods, Rasters

Usage

Cache(FUN, ..., notOlderThan = NULL, objects = NULL, outputObjects = NULL,
  algo = "xxhash64", cacheRepo = NULL, compareRasterFileLength = 1e+06,
  userTags = c(), digestPathContent = TRUE, omitArgs = NULL,
  classOptions = list(), debugCache = character(), sideEffect = FALSE,
  makeCopy = FALSE, quick = FALSE)

# S4 method for ANY Cache(FUN, ..., notOlderThan = NULL, objects = NULL, outputObjects = NULL, algo = "xxhash64", cacheRepo = NULL, compareRasterFileLength = 1e+06, userTags = c(), digestPathContent = TRUE, omitArgs = NULL, classOptions = list(), debugCache = character(), sideEffect = FALSE, makeCopy = FALSE, quick = FALSE)

Arguments

FUN

Either a function or an unevaluated function call (e.g., using quote.

...

Arguments of FUN function .

notOlderThan

load an artifact from the database only if it was created after notOlderThan.

objects

Character vector of objects to be digested. This is only applicable if there is a list, environment or simList with named objects within it. Only this/these objects will be considered for caching, i.e., only use a subset of the list, environment or simList objects.

outputObjects

Optional character vector indicating which objects to return. This is only relevant for simList objects

algo

The algorithms to be used; currently available choices are md5, which is also the default, sha1, crc32, sha256, sha512, xxhash32, xxhash64 and murmur32.

cacheRepo

A repository used for storing cached objects. This is optional if Cache is used inside a SpaDES module.

compareRasterFileLength

Numeric. Optional. When there are Rasters, that have file-backed storage, this is passed to the length arg in digest when determining if the Raster file is already in the database. Note: uses digest for file-backed Raster. Default 1e6. Passed to .prepareFileBackedRaster.

userTags

A character vector with Tags. These Tags will be added to the repository along with the artifact.

digestPathContent

Logical. Should arguments that are of class Path (see examples below) have their name digested (FALSE), or their file contents (TRUE; default).

omitArgs

Optional character string of arguments in the FUN to omit from the digest.

classOptions

Optional list. This will pass into .robustDigest for specific classes. Should be options that the .robustDigest knows what to do with.

debugCache

Character or Logical. Either "complete" or "quick" (uses partial matching, so "c" or "q" work). TRUE is equivalent to "complete". If "complete", then the returned object from the Cache function will have two attributes, debugCache1 and debugCache2, which are the entire list(...) and that same object, but after all .robustDigest calls, at the moment that it is digested using fastdigest, respectively. This attr(mySimOut, "debugCache2") can then be compared to a subsequent call and individual items within the object attr(mySimOut, "debugCache1") can be compared. If "quick", then it will return the same two objects directly, without evalutating the FUN(...).

sideEffect

Logical or path. Deteremines where the function will look for new files following function completion. See Details. NOTE: this argument is experimental and may change in future releases.

makeCopy

Logical. If sideEffect = TRUE, and makeCopy = TRUE, a copy of the downloaded files will be made and stored in the cacheRepo to speed up subsequent file recovery in the case where the original copy of the downloaded files are corrupted or missing. Currently only works when set to TRUE during the first run of Cache. Default is FALSE. NOTE: this argument is experimental and may change in future releases.

quick

Logical. If sideEffect = TRUE, setting this to TRUE, will hash the file's metadata (i.e., filename and file size) instead of hashing the contents of the file(s). If set to FALSE (default), the contents of the file(s) are hashed. NOTE: this argument is experimental and may change in future releases.

Value

As with cache, returns the value of the function call or the cached version (i.e., the result from a previous call to this same cached function with identical arguments).

Filepaths

If a function has a path argument, there is some ambiguity about what should be done. Possibilities include:

  1. hash the string as is (this will be very system specific, meaning a Cache call will not work if copied between systems or directories);

  2. hash the basename(path);

  3. hash the contents of the file.

If paths are passed in as is (i.e,. character string), the result will not be predictable. Instead, one should use the wrapper function asPath(path), which sets the class of the string to a Path, and one should decide whether one wants to digest the content of the file (using digestPathContent = TRUE), or just the filename ((digestPathContent = FALSE)). See examples.

Stochasticity

In general, it is expected that caching will only be used when stochasticity is not relevant, or if a user has achieved sufficient stochasticity (e.g., via sufficient number of calls to experiment) such that no new explorations of stochastic outcomes are required. It will also be very useful in a reproducible workflow.

<code>sideEffect</code>

If sideEffect is not FALSE, then metadata about any files that added to sideEffect will be added as an attribute to the cached copy. Subsequent calls to this function will assess for the presence of the new files in the sideEffect location. If the files are identical (quick = FALSE) or their file size is identical (quick = TRUE), then the cached copy of the function will be returned (and no files changed). If there are missing or incorrect files, then the function will re-run. This will accommodate the situation where the function call is identical, but somehow the side effect files were modified. If sideEffect is logical, then the function will check the cacheRepo; if it is a path, then it will check the path. The function will assess whether the files to be downloaded are found locally prior to download. If it fails the local test, then it will try to recover from a local copy if (makeCopy had been set to TRUE the first time the function was run. Currently, local recovery will only work ifmakeCOpy was set to TRUE the first time Cache was run). Default is FALSE.

Details

Caching R objects using cache has four important limitations:

  1. the archivist package detects different environments as different;

  2. it also does not detect S4 methods correctly due to method inheritance;

  3. it does not detect objects that have file-base storage of information (specifically RasterLayer-class objects);

  4. the default hashing algorithm is relatively slow.

This version of the Cache function accommodates those four special, though quite common, cases by:

  1. converting any environments into list equivalents;

  2. identifying the dispatched S4 method (including those made through inheritance) before hashing so the correct method is being cached;

  3. by hashing the linked file, rather than the Raster object. Currently, only file-backed Raster* objects are digested (e.g., not ff objects, or any other R object where the data are on disk instead of in RAM);

  4. using fastdigest internally when the object is in RAM, which can be up to ten times faster than digest. Note that file-backed objects are still hashed using digest.

If Cache is called within a SpaDES module, then the cached entry will automatically get 3 extra userTags: eventTime, eventType, and moduleName. These can then be used in clearCache to selectively remove cached objects by eventTime, eventType or moduleName.

Cache will add a tag to the artifact in the database called accessed, which will assign the time that it was accessed, either read or write. That way, artifacts can be shown (using showCache) or removed (using clearCache) selectively, based on their access dates, rather than only by their creation dates. See example in clearCache. Cache (uppercase C) is used here so that it is not confused with, and does not mask, the archivist::cache function.

See Also

cache, .robustDigest

Examples

Run this code
# NOT RUN {
library(raster)

tmpDir <- file.path(tempdir(), "reproducible_examples", "Cache")
try(clearCache(tmpDir), silent = TRUE) # just to make sure it is clear

# Basic use
ranNumsA <- Cache(rnorm, 10, 16, cacheRepo = tmpDir)

# All same
ranNumsB <- Cache(rnorm, 10, 16, cacheRepo = tmpDir) # recovers cached copy
ranNumsC <- rnorm(10, 16) %>% Cache(cacheRepo = tmpDir) # recovers cached copy
ranNumsD <- Cache(quote(rnorm(n = 10, 16)), cacheRepo = tmpDir) # recovers cached copy

# Any minor change makes it different
ranNumsE <- rnorm(10, 6) %>% Cache(cacheRepo = tmpDir) # different

## Example 1: basic cache use with tags
ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:b")

showCache(tmpDir, userTags = c("objectName"))
showCache(tmpDir, userTags = c("^a$")) # regular expression ... "a" exactly
showCache(tmpDir, userTags = c("runif")) # show only cached objects made during runif call

clearCache(tmpDir, userTags = c("runif")) # remove only cached objects made during runif call
showCache(tmpDir) # only those made during rnorm call

clearCache(tmpDir)

## Example 2: using the "accessed" tag
ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:b")

# access it again, from Cache
ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
wholeCache <- showCache(tmpDir)

# keep only items accessed "recently" (i.e., only objectName:a)
onlyRecentlyAccessed <- showCache(tmpDir, userTags = max(wholeCache[tagKey == "accessed"]$tagValue))

# inverse join with 2 data.tables ... using: a[!b]
# i.e., return all of wholeCache that was not recently accessed
toRemove <- unique(wholeCache[!onlyRecentlyAccessed], by = "artifact")$artifact
clearCache(tmpDir, toRemove) # remove ones not recently accessed
showCache(tmpDir) # still has more recently accessed

clearCache(tmpDir)

## Example 3: using keepCache
ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:b")

# keep only those cached items from the last 24 hours
oneDay <- 60 * 60 * 24
keepCache(tmpDir, after = Sys.time() - oneDay)

# Keep all Cache items created with an rnorm() call
keepCache(tmpDir, userTags = "rnorm")

# Remove all Cache items that happened within a rnorm() call
clearCache(tmpDir, userTags = "rnorm")

showCache(tmpDir) ## empty

## Example 4: searching for multiple objects in the cache

# default userTags is "and" matching; for "or" matching use |
ranNumsA <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:b")

# show all objects (runif and rnorm in this case)
showCache(tmpDir)

# show objects that are both runif and rnorm
# (i.e., none in this case, because objecs are either or, not both)
showCache(tmpDir, userTags = c("runif", "rnorm")) ## empty

# show objects that are either runif or rnorm ("or" search)
showCache(tmpDir, userTags = "runif|rnorm")

# keep only objects that are either runif or rnorm ("or" search)
keepCache(tmpDir, userTags = "runif|rnorm")

clearCache(tmpDir)

## Example 5: using caching to speed up rerunning expensive computations
ras <- raster(extent(0, 10, 0, 10), res = 1,
              vals = sample(1:5, replace = TRUE, size = 1e2),
              crs = "+proj=lcc +lat_1=48 +lat_2=33 +lon_0=-100 +ellps=WGS84")

# A slow operation, like GIS operation
notCached <- suppressWarnings(
  # project raster generates warnings when run non-interactively
  projectRaster(ras, crs = crs(ras), res = 5, cacheRepo = tmpDir)
)

cached <- suppressWarnings(
  # project raster generates warnings when run non-interactively
  Cache(projectRaster, ras, crs = crs(ras), res = 5, cacheRepo = tmpDir)
)

# second time is much faster
reRun <- suppressWarnings(
  # project raster generates warnings when run non-interactively
  Cache(projectRaster, ras, crs = crs(ras), res = 5, cacheRepo = tmpDir)
)

# recovered cached version is same as non-cached version
all.equal(notCached, reRun) ## TRUE

## Example 6: working with file paths

# if passing a character string, it will take 2 complete passes to before
#  a cached copy is used when it is a save event (read or load is different)
obj <- 1:10
fname <- tempfile(fileext = ".RData")
Cache(saveRDS, obj, file = fname, cacheRepo = tmpDir)
Cache(saveRDS, obj, file = fname, cacheRepo = tmpDir)
Cache(saveRDS, obj, file = fname, cacheRepo = tmpDir) # cached copy is loaded

# however, using asPath(), cached version retrieved after being run once
fname1 <- tempfile(fileext = ".RData")
Cache(saveRDS, obj, file = asPath(fname1), cacheRepo = tmpDir)
Cache(saveRDS, obj, file = asPath(fname1), cacheRepo = tmpDir) # cached copy is loaded

clearCache(tmpDir)

## cleanup
unlink(c("filename.rda", "filename1.rda"))
unlink(dirname(tmpDir), recursive = TRUE)
# }

Run the code above in your browser using DataLab