spatial.tools (version 1.4.8)

focal_hpc: Engine for performing fast, easy-to-develop pixel and focal raster calculations with parallel processing capability.

Description

Engine for performing fast, easy-to-develop pixel and focal raster calculations with parallel processing capability.

Usage

focal_hpc(x, fun, args = NULL, window_dims = c(1, 1),
  window_center = c(ceiling(window_dims[1]/2), ceiling(window_dims[2]/2)),
  filename = NULL, overwrite = FALSE, outformat = "raster",
  additional_header = "ENVI", datatype = "FLT8S", processing_unit = NULL,
  chunk_format = "array", minblocks = "max", blocksize = NULL,
  outbands = NULL, outfiles = NULL, setMinMax = FALSE,
  debugmode = FALSE, .packages = NULL, clearworkers = TRUE,
  verbose = FALSE, ...)

Arguments

x

Raster*. A Raster* used as the input into the function. Multiple inputs should be stack()'ed or list()'ed together.

fun

function. A focal function to be applied to the image. See Details.

args

list. Arguments to pass to the function (see ?mapply). Note that the 'fun' should explicitly name the variables.

window_dims

Vector. The size of a processing window in col x row order. Be default, a single pixel (c(1,1).

window_center

Vector. The local coordinate of the center of a processing window. By default the middle of the processing window. UNSUPPORTED.

chunk_format

Character. The format to send the chunk to the function. Can be "array" (default) or "raster".

minblocks

Numeric. The minimum number of chunks to divide the raster into for processing. Defaults to 1.

blocksize

Numeric. The size (in rows) for a block of data. If unset, focal_hpc will attempt to figure out an optimal blocksize.

filename

Character. Filename(s) of the output raster.

outformat

Character. Outformat of the raster. Must be a format usable by hdr(). Default is 'raster'. CURRENTLY UNSUPPORTED.

overwrite

Logical. Allow files to be overwritten? Default is FALSE.

processing_unit

Character. ("single"|"chunk") Will be auto-set if not specified ("chunk" for pixel-processing, "single" for focal processing). See Description.

outbands

Numeric. If known, how many bands in each output file? Assigning this and outfiles will allow focal_hpc to skip the pre-check.

outfiles

Numeric. If known, how many output files? Assigning this and outbands will allow focal_hpc to skip the pre-check.

setMinMax

Logical. Run a setMinMax() on each output file after processing (this will slow the processing down). Default is FALSE.

additional_header

Character. Create additional output headers for use with other GIS systems (see hdr). Set to NULL to suppress. Default is "ENVI".

datatype

Character. Output number type. See ?dataType. Default is "FLT8S".

debugmode

Logical or Numeric. If TRUE or 1, the function will enter debug mode during the test phase. If debugmode equals 2, the function will stop after the test phase, but won't explicitly enter debug mode. This is useful if the user function has a browser() statement within it. Note the inputs will be an array of size 2 columns, 1 row, and how ever many input bands.

.packages

Character. A character vector of package names needed by the function (parallel mode only).

clearworkers

Logical. Force the workers to clear all objects upon completing (releasing memory)? Default=TRUE.

verbose

logical. Enable verbose execution? Default is FALSE.

...

Additional parameters (none at present).

Details

focal_hpc is designed to execute a function on a Raster* object using foreach, to achieve parallel reads, executions and writes. Parallel random writes are achieved through the use of mmap, so individual image chunks can finish and write their outputs without having to wait for all nodes in the cluster to finish and then perform sequential writing. On Windows systems, random writes are possible but apparently not parallel writes. focal_hpc solves this by trying to write to a portion of the image file, and if it finds an error (a race condition occurs), it will simply retry the writes until it successfully finishes. On Unix-alikes, truly parallel writes should be possible.

Note that rasterEngine is a convienence wrapper for focal_hpc and, in general, should be used instead of focal_hpc directly.

focal_hpc operates in two modes, which have different input and outputs to the function:

Pixel based processing:

1) If chunk_format=="array" (default), the input to the function should assume an array of dimensions x,y,z where x = the number of columns in a chunk, y = the number of rows in the chunk, and z = the number of bands in the chunk. If chunk_format=="raster", the input to the function will be a raster subset. Note that we are ordering the array using standards for geographic data, (columns, rows, bands), not how R usually thinks of arrays (rows, columns, bands).

2) The output of the function should always be an array with the x and y dimensions matching the input, and an arbitrary number of band outputs. Remember to order the dimensions as columns, rows, bands (x,y,z).

Local window processing:

1) The function should be written to process a SINGLE window at a time, given the dimensions of window_dims, so the input to the function should assume a window of dimensions window_dims with a local center defined by window_center. As with before, the input can be passed to the function as an array (suggested) or a small raster.

2) The output should be a single pixel value, so can either be a single value, or a vector (which is assumed to be multiple bands of a single pixel).

The speed of the execution when running in parallel will vary based on the specific setup, and may, indeed, be slower than a sequential execution (e.g. with calc() ), particularly on smaller files. Note that by simply running sfQuickStop(), focal_hpc will run in sequential mode.

See Also

rasterEngine, foreach, mmap, dataType, hdr

Examples

# NOT RUN {
tahoe_highrez <- brick(system.file("external/tahoe_highrez.tif", package="spatial.tools"))
# Pixel-based processing:
ndvi_function <- function(x)
{
	# Note that x is received by the function as a 3-d array:
	red_band <- x[,,2]
	nir_band <- x[,,3]
	ndvi <- (nir_band - red_band)/(nir_band + red_band)
	# The output of the function should also be a 3-d array,
	# even if it is a single band:
	ndvi <- array(ndvi,dim=c(dim(x)[1],dim(x)[2],1))
	return(ndvi)
}

sfQuickInit(cpus=2)
tahoe_ndvi <- focal_hpc(x=tahoe_highrez,fun=ndvi_function)
sfQuickStop()

# }
# NOT RUN {
# Focal-based processing:
local_smoother <- function(x)
{
	# Assumes a 3-d array representing
	# a single local window, and return
	# a single value or a vector of values.
	smoothed <- apply(x,3,mean)
	return(smoothed)
}

# Apply the function to a 3x3 window:
sfQuickInit(cpus=2)
tahoe_3x3_smoothed <- focal_hpc(x=tahoe_highrez,fun=local_smoother,window_dims=c(3,3))
sfQuickStop()

# Example with 7 x 7 window in full parallel mode:
sfQuickInit()
tahoe_7x7_smoothed <- focal_hpc(x=tahoe_highrez,fun=local_smoother,window_dims=c(7,7))
sfQuickStop()
# }