Function that returns an R object with observations corresponding to spikes replaced by values computed from neighboring pixels. Spikes are values in spectra that are unusually high compared to neighbors. They are usually individual values or very short runs of similar "unusual" values.
despike(
x,
height.threshold,
z.threshold,
k,
spike.direction,
window.width,
method,
na.rm,
max.spike.width,
...
)# S3 method for default
despike(
x,
height.threshold,
z.threshold = NA,
k = NA,
spike.direction = NA,
window.width = NA,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...
)
# S3 method for numeric
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...
)
# S3 method for data.frame
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...,
y.var.name = NULL,
var.name = y.var.name
)
# S3 method for generic_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...,
y.var.name = NULL,
var.name = y.var.name
)
# S3 method for source_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
unit.out = getOption("photobiology.radiation.unit", default = "energy"),
...
)
# S3 method for response_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
unit.out = getOption("photobiology.radiation.unit", default = "energy"),
...
)
# S3 method for filter_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
filter.qty = getOption("photobiology.filter.qty", default = "transmittance"),
...
)
# S3 method for reflector_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...
)
# S3 method for solute_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...
)
# S3 method for cps_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...
)
# S3 method for raw_spct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...
)
# S3 method for generic_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...,
y.var.name = NULL,
var.name = y.var.name,
.parallel = FALSE,
.paropts = NULL
)
# S3 method for source_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
unit.out = getOption("photobiology.radiation.unit", default = "energy"),
...,
.parallel = FALSE,
.paropts = NULL
)
# S3 method for response_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
unit.out = getOption("photobiology.radiation.unit", default = "energy"),
...,
.parallel = FALSE,
.paropts = NULL
)
# S3 method for filter_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
filter.qty = getOption("photobiology.filter.qty", default = "transmittance"),
...,
.parallel = FALSE,
.paropts = NULL
)
# S3 method for reflector_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...,
.parallel = FALSE,
.paropts = NULL
)
# S3 method for solute_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...,
.parallel = FALSE,
.paropts = NULL
)
# S3 method for cps_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...,
.parallel = FALSE,
.paropts = NULL
)
# S3 method for raw_mspct
despike(
x,
height.threshold = 10,
z.threshold = 5,
k = 20,
spike.direction = "both",
window.width = 11,
method = "run.mean",
na.rm = FALSE,
max.spike.width = NA,
...,
.parallel = FALSE,
.paropts = NULL
)
A copy of the object passed as argument to x with values
detected as spikes replaced by a local average of neighbours
outside the spike.
numeric vector containing the data.
numeric The minimum height of spikes expressed
relative to the median amplitude of the baseline local variation of
x.
numeric Modified local \(Z\) values larger than
z.threshold are detected as boundaries of spikes.
integer width of median window used for smoothing; must be odd
character Controls the direction of spikes to be
detected. Accepted arguments are "up", "down",
"both".
integer. The full width of the window used for the running mean.
character The name of the method: "run.mean" is running
mean as described in Whitaker and Hayes (2018); "adj.mean" is mean
of adjacent neighbors (isolated bad pixels only).
logical indicating whether NA values should be stripped
before searching for spikes.
integer The width of the widest spike to be detected,
NA puts no limit.
passed in recursive calls.
character Names of columns where to look for spikes to remove.
character One of "energy" or "photon"
character One of "transmittance" or "absorbance"
if TRUE, apply function in parallel, using parallel backend provided by foreach
a list of additional options passed into the foreach function when parallel computation is enabled. This is important if (for example) your code relies on external data or packages: use the .export and .packages arguments to supply them so that all cluster nodes have the correct environment set up for computing.
Spikes are detected based on a modified \(Z\) score calculated from the differenced spectrum. The \(Z\) threshold used should be adjusted to the characteristics of the input and desired sensitivity. The lower the threshold the more stringent the test becomes, with shorter spikes being detected.
The algorithms assume a consistent step size for the underlying
independent variable, e.g., wavelength or time, and should not be applied
if the data do not fulfil this assumption, at least approximately. As
find_spkikes() operates on a single vector, checking this remains
the responsibility of calling functions or methods such as
spikes() and despike().
The algorithm uses running differences to detect abrupt changes in value, compared to an estimate of the baseline variation of the differences, approximating a baseline \(Z\) from MAD and a baseline value from the median differences. Currently, a single estimate of MAD is used but running medians, when possible, as baseline. This comparison detects running differences that are unusually large, in most cases signalling a transition between values near the baseline and far from it, in both directions.
Transitions into- and out of spikes are distinguished based on the median of the non-differenced values, as a descriptor of the data baseline. As for the median of the differences, a running median is used when possible.
This function thus detects the start and end of each spike, and distinguishes upward and downward spikes.
k is the width in number of observations of the window used for
running median smoothing to extract the baseline. A value several times the
width of the broader spike but narrow enough to track broader peaks needs
to be manually set in most cases.
With na.rm = TRUE, NA values are omitted before searching for
spikes and set to 0L in the returned vector.
If all spikes are guaranteed to be one observation-wide and either going up
or down from the baseline, it is possible to detect them based purely on
the z.threshold by passing height.threshold = NA and either
spike.direction = "up" or spike.direction = "down", which
ensures very fast computation.
Parameters of the algorithm need to be adjusted depending on the data, so
inspection of returned values is needed together with adjustment by trial
and error of suitable values for z.threshold,
height.threshold, and k.
Parameter max.spike.width searches for too wide spikes in the
output of the algorithms described above and ignores them. This is
possibly redundant, but maintained for partial backwards compatibility.
Simple interpolation enabled by method = "adj.mean" replaces values of
isolated bad pixels by the mean of their two closest neighbours. The running
mean approach enabled by method = "run.mean" allows the replacement of
short runs of bad pixels by the running mean of neighboring pixels within a
window of user-specified width. The first approach works well for spectra
from array spectrometers to correct for hot and dead pixels in an instrument.
The second approach is most suitable for Raman spectra in which spikes
triggered by radiation are wider than a single pixel but usually not more
than five pixels wide.
Simple interpolation can replace spikes at any position in x, using
a single neighbour as replacement at the extremes of x instead of the
mean of two neighbours. The
running mean approach does not replace those pixels whose distance to the
first or last member of x is less than half the window used for the
running mean, issuing a warning.
When na.rm = TRUE, NA values are considered "bad pixels" and
replaced as such rather than discarded with no replacement. This is the
default behaviour.
See find_spikes() for locating spikes in a vector,
spikes() for extracting/detecting spikes in spectra and
and replace_bad_pixs() for replacing by interpolation
missing or bad values in a vector.
white_led.raw_spct[120:125, ]
# find and replace spike at 245.93 nm
despike(white_led.raw_spct,
z.threshold = 5,
window.width = 7)[120:125, ]
# A high z.threshold value detects more extreme spikes
despike(white_led.raw_spct,
z.threshold = 50,
window.width = 7)[120:125, ]
Run the code above in your browser using DataLab