flashlighttext (version 0.1.0)

build_trie: Trie Builder

Description

Helper to build a trie with a Dictionary.

Usage

build_trie(lm, token_dict, lexicon, word_dict, separator_idx)

Value

a populated Trie instance.

Arguments

lm

a KenLM instance

token_dict

a DIctionary instance with the tokens.

lexicon

a named list with the words and its tokens.

word_dict

a Dictionary instance with the words.

separator_idx

an integer. The index of the separator token.

Examples

library(flashlighttext)
lm <- ZeroLM$new()
lexicon <- list(
  "hello" = list(c("h", "e", "l", "l", "o", "|")),
  "world" = list(c("w", "o", "r", "l", "d", "|")),
  "" = list()
)
word_dict <- create_word_dict(lexicon)
token_dict <- Dictionary$new(c("h", "e", "l", "o", "|", "w", "r", "d", "")) 
token_dict$add_entry("<1>")
trie <- build_trie(
  lm = lm,
  token_dict = token_dict,
  lexicon = lexicon, 
  word_dict = word_dict,
  separator_idx = token_dict$get_index("|")
)