powered by
Apply BPE Merges
apply_bpe(tokens, merge_ranks)
Character vector after BPE merges
Character vector of tokens
Named vector of merge rankings