text2map (version 0.2.3)

test_anchors: Evaluate anchor sets in defining semantic relations

Description

This function evaluates how well an anchor set defines a semantic relations using one of two methods: pairdir (which only evaluates semantic directions) or relco which evaluations semantic directions, semantic centroids and compound concepts). See details.

Usage

test_anchors(
  anchors,
  wv,
  non_anchors = NULL,
  method,
  all = FALSE,
  type = c("direction", "centroid", "compound"),
  conf = 0.95,
  dir_method = c("paired", "pooled", "L2", "PCA"),
  n_runs = 100,
  null = 0,
  alpha = 0.5,
  seed = NULL,
  order_non_anchors = FALSE,
  summarize = TRUE
)

Value

dataframe or list

Arguments

anchors

A data frame or list of 'anchor' terms

wv

Matrix of word embedding vectors (a.k.a embedding model) with rows as terms.

non_anchors

For 'relco', terms that are not anchors (random, unrelated, or distinctive terms).

method

Which metric used to evaluate, 'pairdir' or 'relco'

all

Logical (default FALSE). Whether to evaluate all possible pairwise combinations of two sets of anchors. If FALSE only the input pairs are used in evaluation and anchor sets must be of equal lengths.

type

For 'relco', indicate which kind of relation, "direction", "centroid", "compound"

conf

For 'relco', confidence interval

dir_method

For 'relco' and type = "direction", indicate the method for calculating direction ("paired", "pooled", "L2", "PCA"), See get_direction() for details.

n_runs

For 'relco', number of runs

null

For 'relco', null hypothesis, default is 0.

alpha

For 'relco', significance level

seed

For 'relco', set sampling seed

order_non_anchors

Logical (default FALSE). For 'relco', should the order of the non-anchor terms be fixed between each run

summarize

Logical (default TRUE). Returns a dataframe with AVERAGE scores for input pairs along with each pairs' contribution. If summarize = FALSE, returns a list with each offset matrix, each contribution, and the average score.

Details

PairDir evaluates how parallel two anchor sets are when used to define a semantic direction. According to Boutyline and Johnston (2023):

"We find that PairDir -- a measure of parallelism between the offset vectors (and thus of the internal reliability of the estimated relation) -- consistently outperforms other reliability metrics in explaining axis accuracy."

Boutyline and Johnston only consider analyst specified pairs. However, if all = TRUE, all pairwise combinations of terms between each set are evaluated. This can allow for unequal sets of anchors, however this increases computational complexity considerably.

Relco (anchor reliability coefficient) evaluates how well individual anchors index a given semantic relation in comparison to a set of non-anchor words. This can be used on semantic directions, semantic relations, or compound concepts. See Taylor et al (2025) for details; see also the CMDist() function.

References

Boutyline, Andrei, and Ethan Johnston. 2023. “Forging Better Axes: Evaluating and Improving the Measurement of Semantic Dimensions in Word Embeddings.” tools:::Rd_expr_doi("10.31235/osf.io/576h3")

Taylor, Marshall, et al. 2025. "A Simulation-Based Slope Metric for Anchor List Reliability in Word Embedding Spaces." tools:::Rd_expr_doi("10.31235/osf.io/sc2ub_v3")

Examples

# load example word embeddings
data(ft_wv_sample)

df_anchors <- data.frame(
  a = c("rest", "rested", "stay", "stand"),
  z = c("coming", "embarked", "fast", "move")
)

# test pairdir
test_anchors(df_anchors, ft_wv_sample, method = "pairdir")
test_anchors(df_anchors, ft_wv_sample, method = "pairdir", all = TRUE)

# test relco
non_anchors <- c("writ", "alloys", "ills", "atlas", "saturn", "cape", "unfolds")
## centroid
test_anchors(df_anchors[, 1], ft_wv_sample, method = "relco", 
             type = "centroid", non_anchors = non_anchors)
## compound
test_anchors(df_anchors$a, ft_wv_sample, method = "relco", 
             type = "compound", non_anchors = non_anchors)
## direction
test_anchors(df_anchors, ft_wv_sample, method = "relco", 
             type = "direction", dir_method = "paired", 
             non_anchors = non_anchors)
test_anchors(df_anchors, ft_wv_sample, method = "relco", 
             type = "direction", dir_method = "pooled", 
             non_anchors = non_anchors)