agnes is fully described in chapter 5 of Kaufman and Rousseeuw (1990).
Compared to other agglomerative clustering methods such as hclust,
agnes has the following features: (a) it yields the
agglomerative coefficient (see agnes.object)
which measures the amount of clustering structure found; and (b)
apart from the usual tree it also provides the banner, a novel
graphical display (see plot.agnes). The agnes-algorithm constructs a hierarchy of clusterings.
At first, each observation is a small cluster by itself. Clusters are
merged until only one large cluster remains which contains all the
observations. At each stage the two nearest clusters are combined
to form one larger cluster.
For method="average", the distance between two clusters is the
average of the dissimilarities between the points in one cluster and the
points in the other cluster.
In method="single", we use the smallest dissimilarity between a
point in the first cluster and a point in the second cluster (nearest
neighbor method).
When method="complete", we use the largest dissimilarity
between a point in the first cluster and a point in the second cluster
(furthest neighbor method).
The method = "flexible" allows (and requires) more details:
The Lance-Williams formula specifies how dissimilarities are
computed when clusters are agglomerated (equation (32) in K.&R.,
p.237). If clusters $C_1$ and $C_2$ are agglomerated into a
new cluster, the dissimilarity between their union and another
cluster $Q$ is given by
$$D(C_1 \cup C_2, Q) = \alpha_1 * D(C_1, Q) + \alpha_2 * D(C_2, Q) +
\beta * D(C_1,C_2) + \gamma * |D(C_1, Q) - D(C_2, Q)|,$$
where the four coefficients $(\alpha_1, \alpha_2, \beta, \gamma)$
are specified by the vector par.method:
If par.method is of length 1,
say $= \alpha$, par.method is extended to
give the Flexible Strategy (K. & R., p.236 f) with
Lance-Williams coefficients $(\alpha_1 = \alpha_2 = \alpha, \beta =
1 - 2\alpha, \gamma=0)$.
If of length 3, $\gamma = 0$ is used.
Care and expertise is probably needed when using method
= "flexible" particularly for the case when par.method is
specified of longer length than one.
The weighted average (method="weighted") is the same as
method="flexible", par.method = 0.5.