The tip labels of tree (accessed via tree$tip.label) must have the same names (and the same length) as the tip labels in unsorted.pvalues, but may be in a different order. The p-values in column 2 of unsorted.pvalues obviously must be in the [0, 1] range. p.cutoffs takes values in the (0, 1) range. The default value for p.cutoffs is c(0.01, 0.05, 0.1, 0.9, 0.95, 0.99) if side is 1 and c(0.01, 0.05, 0.1) if side is 2. Thus, the ranges (when side is 1) are: [0, .01], (.01, .05], ..., (.99, 1]. These ranges correspond to the colors specified in pal. P-values in the [0, .01] range correspond to the left-most color if pal is a palette (view this via display.brewer.pal(x, pal) - where x is the number of colors to be used) or the first value in the vector if pal is a vector of colors. If pal is a vector of colors, then the length of pal should be one greater than the length of p.cutoffs. In other words, its length must be the same as the number of p-value ranges. In addition, each color in this vector of colors needs to be in hexadecimal format, for example, "#B2182B". Formats of colors other than hexadecimal will likely give unwanted results in the edges of the tree produced in FigTree, such as all-black edges or the edges being colored in a meaningless way. This is because the color conversion assumes hexadecimal colors. The default value of pal is "RdBu" (a divergent palette of reds and blues, with reds corresponding to small p-values) if side is 1 and the reverse of "Reds" (a sequential palette) if side is 2. The sequential palettes in RColorBrewer go from light to dark, so "Reds" is reversed so that the dark red corresponds to small p-values. It probably makes more sense to use a divergent palette when using 1-sided p-values and a sequential palette (reversed) when using 2-sided p-values. To create a vector of reversed colors from a palette with x number of colors and "PaletteName" as the name of the palette, use rev(brewer.pal(x, "PaletteName")). ignore.edge.length may be useful to get a more uniformly-shaped tree. export.figtree assumes that each internal node has exactly two descendants. It also assumes that each internal node has a lower number than each of its ancestors (excluding tips).
The branch argument controls whether edge coloring corresponds to the combined p-value of the tips below the edge ("edge") or of the tips below the edge's leading (away from the tips) node ("node"). Note that if branch="node" is used, then both edges leaving a node will necessarily be colored the same.
To access the tutorial document for this package (including this function), type in R: vignette("SigTree")