"tree" returns the vector "x" and imposes on it two attributes: name and shape. The name attribute is used in the subroutine plotpersp; the shape attribute is "tree", and according to the value of the vector itself and its shape attribute, the cone edges of the cone generated by the constraint matrix, which constrains that $\eta$ has a tree-ordering in "x" will be made. The cone edges are a set of basis employed in the hinge algorithm.
Note that "tree" does not make the corresponding cone edges itself. It sets things up to a sub-routine called tree.fun in cgam which will make the cone edges. A tree-ordering is a partial ordering: For a categorical variable $x$, if there are treatment levels $x_1,\ldots,x_k$, where $x_1$ is a placebo, we compare $x_i, i = 2,\ldots,k$ with $x_1$, and not have any other comparable pairs.
See references cited in this section for more details.