UnigramTokenizer
_UnigramCommonMixinTokenizerUnigramTokenizer(pieces: list[tuple[str, float]], unk_token: str = '<unk>', unk_log_prob: float = -100.0, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Reference (pure-Python) Unigram tokenizer.
Encodes via Viterbi over the lattice of all candidate
sub-piece segmentations: for each chunk we build a DP table
dp[i] = best (highest log-prob) path ending at byte offset
i, with single-codepoint UNK fallback when no piece spans
a region. Decode is a trivial "".join(pieces).replace("▁", " ").
Training is delegated to C++ (see train for the
rationale) — even this "pure-Python" flavour does not implement
EM in Python because the runtime would be unusable on any
real corpus.
For production / latency-sensitive use, prefer
UnigramTokenizerFast — same vocab format, bit-identical
encode output, but the Viterbi hot loop runs in C++.
Parameters
pieceslist of (str, float)(piece_str, log_prob) list; index = token id.
Larger (less-negative) log-probs are preferred during
Viterbi decode.unk_tokenstr= "<unk>"pieces
spans a region of the input. Must appear in pieces for
the UNK id to be defined; otherwise encode silently drops
unmatchable codepoints.unk_log_probfloat= -100.0normalizerNormalizer= Nonelucid.utils.tokenizer._normalizers.NFKC (matches
LLaMA / Mistral / T5).pre_tokenizerPreTokenizer= NoneSentencePiecePreTokenizer for canonical
SentencePiece behaviour with ▁ word-start markers.lucid.utils.tokenizer.SpecialTokens. Defaults to
SpecialTokens(unk=unk_token).Notes
The Viterbi DP runs in per chunk, where
N is the chunk byte length and M is the max piece byte
length. UTF-8 boundary masking ensures sub-piece offsets only
land on codepoint boundaries (no mid-codepoint cuts), which
matches the C++ flavour bit-for-bit.
See Also
UnigramTokenizerFast—C++-backed flavour with identicalencode output and a much faster—meth:~UnigramTokenizerFast.encode.
Used by 1
Constructors
1__init__
→None__init__(pieces: list[tuple[str, float]], unk_token: str = '<unk>', unk_log_prob: float = -100.0, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python Unigram tokenizer.
Parameters
pieceslist of (str, float)(piece, log_prob) list; index = token id.unk_tokenstr= "<unk>"unk_log_probfloat= -100.0normalizer(Normalizer or None, optional, keyword - only)= NoneNFKC.pre_tokenizer(PreTokenizer or None, optional, keyword - only)= NoneSentencePiecePreTokenizer.SpecialTokens(unk=unk_token).Notes
Builds piece-id maps and caches the longest piece in bytes
for the Viterbi DP via _rebuild_tables.
Properties
3Algorithm identifier (always "unigram").
Returns
strConstant string "unigram".
Raw (piece, log_prob) list — useful for inspection
and for handing off to UnigramTokenizerFast.
Returns
list of (str, float)Shallow copy of the internal pieces list.
Number of pieces in the Unigram vocabulary.
Returns
intlen(self._pieces).
Class methods
1from_file(directory: str, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Load from a directory containing tokenizer.json.
Parameters
directorystrtokenizer.json (and
optionally special_tokens_map.json).normalizer(Normalizer, optional, keyword - only)= Nonelucid.utils.tokenizer._normalizers.NFKC
when omitted (matches LLaMA / Mistral / T5).pre_tokenizer(PreTokenizer, optional, keyword - only)= NoneSentencePiecePreTokenizer for
canonical SentencePiece behaviour.None (the
default), parsed from special_tokens_map.json if
present.Returns
UnigramTokenizerFreshly-constructed instance ready for encode / decode.
Instance methods
4Return a copy of the piece → id map.
Returns
dict[str, int]Shallow copy; mutating it does not affect the tokenizer.
Look up the piece string for a token id.
Parameters
token_idintReturns
str or NoneThe piece string, or None if token_id is unknown.
Persist as unified tokenizer.json + special_tokens_map.json.
Parameters
directorystrRe-train this tokenizer from scratch on corpus.
Implements the Kudo-2018 EM-with-pruning training loop:
- Pre-tokenize each document (using the configured pre-tokenizer chain) into chunks.
- Seed a large candidate vocab from all sub-strings, then iteratively run EM to estimate per-piece probabilities and prune the lowest-contribution pieces until the target vocab size is reached.
- Re-load the resulting
(piece, log_prob)list into Python state.
Even this "pure-Python" flavour delegates the inner loop to C++ — EM is a tight numerical loop and a Python rewrite would be 100-1000x slower for any non-trivial corpus.
Parameters
corpusiterable of strvocab_sizeint= 30 000