WordPiece tokenizer. Owns a string→id vocab + dense reverse table. Continuation pieces are stored with the ## prefix already baked into the vocab key (BERT convention) — encode/decode are responsible for adding/stripping the prefix at segment boundaries. unk_id_ is cached so the encode hot path never has to hash the UNK string. Matches the Python lucid.utils.tokenizer.WordPieceTokenizer.
Constructors
1Methods
11── Tokenizer overrides ────────────────────────────────────────
Apply greedy longest-match to ONE pre-tokenized word. Returns the matched ids, or a single UNK id if the word cannot be tokenized at any position.
Rebuild reverse table after vocab mutation.
Train on a list of pre-tokenized words (the Python wrapper does the normalization + word-splitting via BertNormalizer + WhitespacePunctuationSplit first). Builds a vocab of size up to target_vocab_size by greedy frequency merging — close to HF's WordPieceTrainer behaviour without the full log-likelihood objective.
── WordPiece-specific accessors ───────────────────────────────