kin.search: Midline tracking over image sequences

Description

Automatically retrieves the midline of a detected ROI in each image of a sequence through thresholding and segmentation; finds the y-value midpoint along the x-value array of the ROI and fits a midline according to a chosen smoothing method (loess or spline). Also outputs the midline amplitude relative to a reference line determined by an anterior section of the ROI. Supported image formats are jpeg, png, and tiff.

Usage

kin.search(
  image.dir = NULL,
  frames = NULL,
  thr = "otsu",
  plot.pml = TRUE,
  show.prog = FALSE,
  ant.per = 0.1,
  tips = 0.02,
  smoothing = "loess",
  smooth = 0.25,
  smooth.points = 200,
  image.type = "orig",
  save = TRUE,
  out.dir = NULL,
  flip = TRUE,
  size.min = 0.02,
  search.for = "largest",
  edges = FALSE,
  border = 5
)

Arguments

image.dir

character, directory containing images to analyze.

frames

numeric, vector indicating which images to process.

thr

numeric or character ('otsu') threshold to determine binary image. See Details.

plot.pml

logical, value indicating if outputted images should include an overlay of the theoretical midline based on ant.per.

show.prog

logical value indicating if outputted image should be displayed during analysis.

ant.per

numeric; left-most percentage of ROI that establishes the horizontal reference for the midline displacement.

tips,

numeric, the proportion the the midline data to use in calculation of the head and tail position.

smoothing

character, the midline smoothing method, either 'loess' or "spline".

smooth

numeric; if smoothing is set to 'loess', smoothing parameter value for plotted midline.

smooth.points

numeric, number of equally spaced points along the ROI midline on which the smoothed midline is computed.

image.type

character; the type of image to be outputted, either 'orig' or 'bin' representing the original or binary images, respectively. Ignored if 'save=FALSE'.

save

logical, value indicating if images should be outputted with midline and predicted midline based on the ant.per lm() overlaying original or binary images.

out.dir

character, the directory to which outputted images should be saved.

flip

logical, indicating if binary should be flipped.

size.min

numeric, indicating the minimum size of ROIs as a proportion of the pixel field to be considered in analysis. May be useful if smaller unimportant ROIs appear in the frame. Default is 0.02.

search.for

character, the search parameter. See Details.

edges

logical, should ROIs on image edges be evaluated. See Details.

border

if edges=TRUE, size of border to add in pixels. Dee details.

Value

A list with the following components:

kin.dat a data table consisting of frame-by-frame position parameters for the ROI determined by search.for.

the frame number
'x' and ''y': the position of the tail (rightmost or posteriormost)
'head.x' and 'head.y': the x and y position of the head (leftmost or anteriormost)
'amp': the amplitude (amp) of the tail relative to thr theoretical midline determined by the lm() predictions from ant.per
'head.pval': p values of the lm() fit that describes the position of the head as determined by ant.per (green points in the outputted images/video)
'roi': a character indicating the ROI ranked by size ('a' being the largest)
'edge': indicating whether ROI was on the edge of the image field
'size': size of the ROI in pixels^2
'offset.x': ROI distance from horizontal center
'offset.y': ROI distance from vertical center
'offset': linear distance of ROI's centroid to image center

midline A data table containing, for each frame described by frames, the following:

'x' and 'y.m': x and y positions of the midline of the ROI #'
'y.min' and 'y.max': min and max y positions ROI's contour used in y.m calculation
'mid.pred': the predicted linear midline based on the points/pixels defined by head.per (green points in the outputted images/video)
'y.pred': midline points fit to a smooth spline or loess model with spar or span equal to smooth (red curve in the outputted images/video)
'wave.y': midline points 'y.pred' relative to 'mid.pred'
'roi': a character indicating ROI size ('a' being the largest)

cont A data table containing x and y positions of the contours used to calculate the data in 'kin.dat'. Contains the following:

'frame': the frame
'x' and 'y': the x and y positions of the contours

all.classes A data table containing the following for all ROIs detected:

'frame': the frame
'roi': the name of each ROI found in a frame.
'edge': indicating whether ROI was on the edge of the image field
'size': size of the ROI in pixels^2
'offset.x': ROI distance from horizontal center
'offset.y': ROI distance from vertical center
'offset': linear distance of ROI's centroid to image center

dim the x and y dimensions of the images analyzed

A list with the following components:

kin.dat a data frame consisting of frame-by-frame position parameters for the ROI indicated by n.blob:

the frame number
'head.x' and 'head.y': the x and y position of the head (leftmost or anteriormost)
'x' and 'y': the position of the tail (rightmost or posteriormost)
'amp': the amplitude (amp) of the tail
'cent.x' and 'cent.y': centroid coordinate of ROI
'roi': a character indicating ROI size ('a' being the largest)
'head.pval': p values of the lm() fit that describes the position of the head as determined by ant.per (green points in the outputted images/video)

midline A data frame containing, for each frame described by frames, the following:

'x' and 'y.m': x and y positions of the midline of the ROI
'mid.pred': the predicted linear midline based on the points/pixels defined by head.per (green points in the outputted images/video)
'y.pred': midline points fit to a smooth spline or loess model with spar or span equal to smooth (red curve in the outputted images/video)
'wave.y': midline points 'y.pred' normalized to 'mid.pred'
'roi': a character indicating ROI size ('a' being the largest)
'cent.x': x centroid of ROI
'cent.y': y centroid of ROI
'offset.x': ROI distance from horizontal center
'offset.y': ROI distance from vertical center
'offset.total': sum of ROI offset.x and offset.y
'ar': aspect ration of the ROI
'size': size of ROI in pixels

dim the x and y dimensions of the images analyzed

Details

The algorithm assumes a left-right orientation, i.e., the head of the ROI is positioned left, the tail right. The ant.per value therefor establishes the reference line (theoretical straight midline) based on that portion of the head. The midline is calculated as the midpoints between the y extrema for each x position. Chooses ROIs based on relative ROI size or position.

Thresholding operations can be performed with an arbitrary (user defined) numeric value or with Otsu's method ('thr="otsu"'). The latter chooses a threshold value by minimizing the combined intra-class variance. See otsu.

If 'edges=TRUE', it is best to add an artificial border so that any part of the ROI in contact with the edge can be distinguished from it.

search.for determines how ROIs are chosen:

"offset", the ROI with a centroid that is the shortest linear distance to the center of the field
"offset.x", the ROI with a centroid x position that is closest to the x position of the center of the field
"offset.y", the ROI with a centroid y position that is closest to the y position of the center of the field
"largest", the largest ROI.

These choices will be made on ROI sets that are not on the edge of the field if 'edges=FALSE'.

edges Set by default to 'FALSE'. It is not advisable to include shapes that are on the edge of any frame and are therefore incomplete. Yet, if set to 'TRUE', the border adds a black border to the image so that the intended ROI may be distinguished from the edge.

image.type Can be set as "orig" or "bin". "orig" plots midline and reference lines over the original video frames, "bin" over binary images.

Examples

Run this code

# NOT RUN {
#### plot lot caudal amplitude and produce a classic midline waveform plot of swimming fish	
##A very long example.
# }
# NOT RUN {
#download example images and place in 'example' subdirectory	
f <- "https://github.com/ckenaley/exampledata/blob/master/example.zip?raw=true"	

download.file(f, paste0(tempdir(),"/temp.zip"))
unzip(paste0(tempdir(),"/temp.zip"), exdir=tempdir())	
unlink(paste0(tempdir(),"/temp.zip"))	

dir.create(paste0(tempdir(),"/processed_images"))
kin <- kin.search(image.dir =paste0(tempdir(),"/example"),	
       search.for = "largest",	
      smoothing = "loess",frames=1:50,
      out.dir=paste0(tempdir(),"/processed_images"),
      show.prog = FALSE,thr = "otsu",	
      image.type="bin",smooth=0.4)	

#plot instantaneous amplitude of tail (last/rightmost point) over frames 	
p <- ggplot(dat=kin$kin.dat,aes(x=frame,y=amp))+geom_line()+geom_point()+theme_classic(15)
print(p)	
	
# midline plot	
ml <- kin$midline	
#leftmost x starts at 0	
ml <- ml[,x2:=x-x[1],by=frame]	

ml <- merge(ml,kin$kin.dat[,list(frame,amp)],by="frame") #merge these	

pal <- wes_palette("Zissou1", 100, type = "continuous") #"Zissou" color palette

p <- ggplot(dat=ml,aes(x=x2,y=wave.y))+theme_classic(15)+scale_color_gradientn(colours = pal)	
p <- p+geom_line(aes(group=frame,color=amp),stat="smooth",method = "loess", size = 1.5)	
print(p)	
	
#Make a video of processed frames	

images.to.video2(image.dir=paste0(tempdir(),"/processed_images"),	
vid.name="trout_test",out.dir=tempdir(),frame.rate=5,qual=100,raw=FALSE)	
file.exists(paste0(tempdir(),"/trout_test_red.mp4"))

# }
# NOT RUN {
## A very short example.

#retrieve image 
i <- EBImage::readImage(system.file("extdata/img", "sunfish_BCF.jpg", package = "trackter"))
#create directory and write image to it
t <- tempdir()


dir.create(paste0(t,"/images"))
EBImage::writeImage(i,paste0(t,"/images/sunfish001.jpg"),type = "jpeg")

list.files(paste0(t,"/images"))
#run kin.search and save output image to directory
kin.i<- kin.search(image.dir = paste0(t,"/images"),smooth=0.7,save = TRUE,out.dir = t)

#plot midline over original image
with(kin.i$midline,plot(x,wave.y))

i2 <- EBImage::readImage(paste0(t,"/sunfish001_000.jpg"))
EBImage::display(i2,method="raster")

#clean up
unlink(paste0(t,"/images"),recursive=TRUE)

# }