morphemepiece-package

<p>Tokenize words into morphemes (the smallest unit of meaning).</p>

Tokenize text into morphemes. The morphemepiece algorithm uses a
lookup table to determine the morpheme breakdown of words, and falls back on a
modified wordpiece tokenization algorithm for words not found in the lookup
table.

Jonathan Bratt

morphemepiece

Morpheme Tokenization

morphemepiece-package: morphemepiece: Morpheme Tokenization

Description

Arguments