UnigramTokenizerFast
_UnigramCommonMixinTokenizerUnigramTokenizerFast(pieces: list[tuple[str, float]], unk_token: str = '<unk>', unk_log_prob: float = -100.0, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)C++-backed Unigram tokenizer.
Identical algorithm + on-disk format to
UnigramTokenizer; the per-chunk Viterbi loop runs in
C++ via the engine Unigram binding. Encode outputs are
bit-identical for the same pieces + same normalizer + same
pre-tokenizer.
Use this in production training / inference. Use
UnigramTokenizer for debugging / extending the
algorithm with custom Python-only normalizers without touching
C++.
Parameters
SameconstructedSee Also
UnigramTokenizer—Pure-Python reference flavour.
Used by 1
Constructors
1__init__
→None__init__(pieces: list[tuple[str, float]], unk_token: str = '<unk>', unk_log_prob: float = -100.0, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed Unigram tokenizer.
Parameters
pieceslist of (str, float)(piece, log_prob) list passed to the C++
backend; index = token id.unk_tokenstr= "<unk>"unk_log_probfloat= -100.0normalizer(Normalizer or None, optional, keyword - only)= NoneNFKC.pre_tokenizer(PreTokenizer or None, optional, keyword - only)= NoneSentencePiecePreTokenizer.SpecialTokens(unk=unk_token).Notes
Constructs the C++ Unigram backend once and caches it
on _cpp; the Python-side _id_to_piece reverse map mirrors
the pieces list for fast decode.
Properties
3Algorithm identifier (always "unigram").
Returns
strConstant string "unigram".
Raw (piece, log_prob) list cached on the Python side.
Returns
list of (str, float)Shallow copy of the cached pieces list (kept in sync with
C++ by train and __init__).
Number of pieces in the live C++ vocabulary.
Returns
intself._cpp.vocab_size().
Class methods
1from_file(directory: str, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Identical loader to UnigramTokenizer.from_file.
The only difference is the returned class (and hence the encode backend — C++ instead of Python Viterbi).
Parameters
directorystrtokenizer.json (and
optionally special_tokens_map.json).normalizer(Normalizer, optional, keyword - only)= Nonelucid.utils.tokenizer._normalizers.NFKC.pre_tokenizer(PreTokenizer, optional, keyword - only)= NoneSentencePiecePreTokenizer.None (the
default), parsed from special_tokens_map.json if
present.Returns
UnigramTokenizerFastFreshly-constructed C++-backed instance ready for encode / decode.
Instance methods
4Return a piece → id map from the C++ backend.
Returns
dict[str, int]Fresh dict built from self._cpp.get_vocab().
Look up the piece string for a token id.
Parameters
token_idintReturns
str or NoneThe piece string, or None if unknown.
Same format as UnigramTokenizer.save.
Re-train in C++ (EM with vocab pruning, Kudo 2018).
Materialises corpus into a list before handing off (the
C++ binding takes std::vector<std::string>); for very
large corpora the caller is responsible for chunking. After
training, the Python-side pieces cache is refreshed from the
C++ side so subsequent encodes see the new state.
Parameters
corpusiterable of strvocab_sizeint= 30 000