load_lookup

<p>path to lookup file. File is assumed to be a text
file, with one word per line. The lookup value, if different from the word,
follows the word on the same line, after a space.</p>

lookup_file

<p>Usually you will want to use the included lookup that can be accessed via
<code>morphemepiece_lookup()</code>. This function can be used to load a different
lookup from a file.</p>

Tokenize text into morphemes. The morphemepiece algorithm uses a
lookup table to determine the morpheme breakdown of words, and falls back on a
modified wordpiece tokenization algorithm for words not found in the lookup
table.

Jonathan Bratt

morphemepiece

load_lookup: Load a morphemepiece lookup file

Description

Usage

Arguments

Value