wordpiece (version 2.1.3)

.process_vocab: Process a Vocabulary for Tokenization

Description

Process a Vocabulary for Tokenization

Usage

.process_vocab(v)
# S3 method for default
.process_vocab(v)
# S3 method for wordpiece_vocabulary
.process_vocab(v)
# S3 method for character
.process_vocab(v)

Arguments

v

An object of class wordpiece_vocabulary or a character vector.

Value

A character vector of tokens for tokenization.