RBERT (version 0.1.11)

WordpieceTokenizer: Construct objects of WordpieceTokenizer class.

Description

(I'm not sure that this object-based approach is best for R implementation, but for now just trying to reproduce python functionality.)

Usage

WordpieceTokenizer(vocab, unk_token = "[UNK]", max_input_chars_per_word = 200)

Arguments

vocab

Recognized vocabulary tokens, as a named integer vector. (Name is token, value is index.)

unk_token

Token to use for unknown words.

max_input_chars_per_word

Length of longest word we will recognize.

Value

an object of class WordpieceTokenizer

Details

Has method: tokenize.WordpieceTokenizer()

Examples

# NOT RUN {
vocab <- load_vocab(vocab_file = "vocab.txt")
wp_tokenizer <- WordpieceTokenizer(vocab)
# }