Shared base for every lookup-based tokenizer. Holds the vocab + reverse table + UNK fallback + the standard encode/decode/train plumbing. Concrete subclasses implement split_to_chunks_.
Constructors
1ctor
LookupTokenizer
void LookupTokenizer()Methods
8Tokenizer overrides — shared across all 5 concretes.
Rebuild reverse table after vocab mutation.
Algorithm-specific pre-tokenization — chop text into the list of chunks that the lookup step then turns into ids.
Default training: pre-tokenize via subclass's split_to_chunks_ + insert each unique chunk into vocab in insertion order until target_vocab_size is reached. ByteTokenizer overrides with a no-op (its vocab is fixed).