class

CharTokenizer

extendsLookupTokenizer

CharTokenizer

── CharTokenizer ───────────────────────────────────────────────────

One token per Unicode codepoint. Vocab is the set of distinct codepoints encountered during training (or supplied at construction time).

Constructors

ctor

void CharTokenizer()

fnconst

int algo()

fnconst

int split_to_chunks_(const int & text)