RBERT (version 0.1.11)

check_vocab: Check Vocabulary

Description

Given some words and a word piece vocabulary, checks to see if the words are in the vocabulary.

Usage

check_vocab(words, ckpt_dir = NULL, vocab_file = find_vocab(ckpt_dir))

Arguments

words

Character vector; words to check.

ckpt_dir

Character; path to checkpoint directory. If specified, any other checkpoint files required by this function (vocab_file, bert_config_file, or init_checkpoint) will default to standard filenames within ckpt_dir.

vocab_file

path to vocabulary file. File is assumed to be a text file, with one token per line, with the line number corresponding to the index of that token in the vocabulary.

Value

A logical vector containing TRUE if the corresponding word was found verbatim in the vocabulary, FALSE otherwise.

Examples

# NOT RUN {
BERT_PRETRAINED_DIR <- download_BERT_checkpoint("bert_base_uncased")
to_check <- c("apple", "appl")
check_vocab(words = to_check, ckpt_dir = BERT_PRETRAINED_DIR) # TRUE, FALSE
#'
# }