WordPieceTokenizer
_WordPieceCommonMixinTokenizerWordPieceTokenizer(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Reference (pure-Python) WordPiece tokenizer.
Implements BERT's WordPiece algorithm in pure Python. Each
pre-tokenized word is split via greedy longest-match: the
longest vocab prefix is emitted, then the algorithm recurses on
the remainder with the continuation prefix ("##" by default)
prepended. If no prefix matches at any position, the whole word
becomes a single UNK token.
Easy to read, easy to step through, no C++ build required. For
production / latency-sensitive use, prefer
WordPieceTokenizerFast — same algorithm, same vocab
format, bit-identical encode outputs, but the hot loop runs in
C++.
Parameters
vocabdict[str, int]##; id = line index when
loaded from vocab.txt.unk_tokenstr= "[UNK]"vocab for it to actually
emit an id (otherwise nothing is emitted for that word).continuing_prefixstr= "##"max_chars_per_wordint= 100BasicTokenizer cap
and bounds worst-case encode cost).normalizerNormalizer= Nonelucid.utils.tokenizer._normalizers.BERTNormalizer.pre_tokenizerPreTokenizer= Nonelucid.utils.tokenizer._pre_tokenizers.WhitespacePunctuationSplit.SpecialTokens(unk=unk_token)
so the tokenizer behaves correctly out of the box.Notes
The greedy longest-match is O(W^2) per word in the worst case
(where W = word length), but typical English words have W < 20,
so the Python interpreter overhead dominates.
WordPieceTokenizerFast recovers an order of magnitude
on long-document encoding.
Examples
>>> tok = WordPieceTokenizer.from_pretrained("bert-base-uncased-dir")
>>> ids = tok.encode("hello world")
>>> tok.decode(ids)
'hello world'See Also
WordPieceTokenizerFast—C++-backed equivalent with the same API.
Used by 3
Constructors
1__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python WordPiece tokenizer.
Parameters
vocabdict[str, int]continuing_prefix.unk_tokenstr= "[UNK]"continuing_prefixstr= "##"max_chars_per_wordint= 100normalizer(Normalizer or None, optional, keyword - only)= NoneBERTNormalizer.pre_tokenizer(PreTokenizer or None, optional, keyword - only)= NoneWhitespacePunctuationSplit.SpecialTokens(unk=unk_token).Properties
2Class methods
1from_file(directory: str, unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Load from a BERT-style directory containing vocab.txt.
Parameters
directorystrvocab.txt and optionally
special_tokens_map.json.unk_token(str, optional, keyword - only)= "[UNK]"continuing_prefix(str, optional, keyword - only)= "##"max_chars_per_word(int, optional, keyword - only)= 100normalizer(Normalizer or None, optional, keyword - only)= Nonepre_tokenizer(PreTokenizer or None, optional, keyword - only)= Nonespecial_tokens_map.json when omitted.Returns
WordPieceTokenizerA new tokenizer populated from disk.
Raises
FileNotFoundErrorvocab.txt is missing.Instance methods
4Return a shallow copy of the token → id map.
Returns
dict[str, int]Copy of the internal vocab.
Look up the surface string for a token id.
Parameters
token_idintReturns
str or NoneThe token string, or None if unknown. Continuation
pieces retain their continuing_prefix (typically
##).
Persist as a BERT-style directory (vocab.txt +
unified tokenizer.json + special_tokens_map.json).
Writing both legacy and unified formats means the resulting
directory loads back via from_pretrained regardless
of which format the caller looks for first.
Parameters
directorystrRe-train this tokenizer from scratch on corpus.
Greedy frequency-based WordPiece training (matches Hugging
Face's WordPieceTrainer semantics; faster than the full
log-likelihood training in the original paper). For each
iteration, counts adjacent symbol-pair frequencies across
all words and merges the most frequent pair, growing the
vocab by 1 each step until vocab_size is reached or no
pair appears more than once.
Replaces _vocab in place; previous contents are
discarded. The result is a BERT-compatible vocab with
## continuation markers and the configured UNK token
reserved at id 0. Tie-break is lexicographic key order for
determinism (same convention as BPE).
Parameters
corpusiterable of strvocab_sizeint= 30 000