grad.desc: Gradient Descent Algorithm for the 2D case.

Description

Gradient Descent Algorithm for the 2D case. This function provids a visual illustration for the process of minimizing a real-valued function through Gradient Descent Algorithm.

Usage

grad.desc(FUN = function(x, y) x^2 + 2 * y^2, rg = c(-3, 
    -3, 3, 3), init = c(-3, 3), gamma = 0.05, tol = 0.001, len = 50, 
    interact = FALSE, col.contour = "red", col.arrow = "blue")

Arguments

FUN

the objective function to be minimized; contains only two independent variables (variable names do not need to be 'x' and 'y')

ranges for independent variables to plot contours; in a c(x0, y0, x1, y1) form

init

starting values

gamma

size of a step

tol

tolerance to stop the iterations, i.e. the minimum difference between $F(x_i)$ and $F(x_{i+1})$

len

desired length of the independent sequences (to compute z values for contours)

interact

logical; whether choose the starting values by cliking on the contour plot directly?

col.contour,col.arrow

colors for the contour lines and arrows respectively (default to be red and blue)

Value

A list containing
parthe solution for the local minimum
valuethe value of the objective function corresponding to par
iterthe number of iterations; if it is equal to control$nmax, it's quite likely that the solution is not reliable because the maximum number of iterations has been reached
gradientthe gradient function of the objective function; it is returned by deriv
perspa function to make the perspective plot of the objective function; can accept further arguments from persp (see the examples below)

Details

Gradient descent is an optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or the approximate gradient) of the function at the current point. If instead one takes steps proportional to the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent. The arrows are indicating the result of iterations and the process of minimization; they will go to a local minimum in the end if the maximum number of iterations (ani.options('nmax') has not been reached.

References

http://en.wikipedia.org/wiki/Gradient_descent http://animation.yihui.name/compstat:gradient_descent_algorithm

Examples

Run this code

## default example
oopt = ani.options(interval = 0.3, nmax = ifelse(interactive(), 
    50, 2))
xx = grad.desc()
xx$par  # solution
xx$persp(col = "lightblue", phi = 30)   # perspective plot

## define more complex functions; a little time-consuming
f1 = function(x, y) x^2 + 3 * sin(y)
xx = grad.desc(f1, pi * c(-2, -2, 2, 2), c(-2 * pi, 
    2))
xx$persp(col = "lightblue", theta = 30, phi = 30)
## or
ani.options(interval = 0, nmax = ifelse(interactive(), 
    200, 2))
f2 = function(x, y) sin(1/2 * x^2 - 1/4 * y^2 + 3) * 
    cos(2 * x + 1 - exp(y))
xx = grad.desc(f2, c(-2, -2, 2, 2), c(-1, 0.5), gamma = 0.1, 
    tol = 1e-04)

## click your mouse to select a start point
if (interactive()) {
    xx = grad.desc(f2, c(-2, -2, 2, 2), interact = TRUE, tol = 1e-04)
    xx$persp(col = "lightblue", theta = 30, phi = 30)
}

## HTML animation pages
saveHTML({
    ani.options(interval = 0.3)
    grad.desc()
}, img.name = "grad.desc", htmlfile = "grad.desc.html", ani.height = 500, 
    ani.width = 500, title = "Demonstration of the Gradient Descent Algorithm", 
    description = "The arrows will take you to the optimum step by step.")

ani.options(oopt)

Run the code above in your browser using DataLab

Get 50% off unlimited learning