Unigram language model tokenizer. Owns an ordered (piece, log_prob) table — entry index doubles as the token id, so the order is part of the on-disk contract and must be preserved across save/load. All encode paths are Viterbi-optimal; the UNK piece is always at a known id so feasibility of segmentation is guaranteed post-train. Matches the Python lucid.utils.tokenizer.UnigramTokenizer wrapper exactly.
Constructors
1Methods
10── Tokenizer overrides ────────────────────────────────────────
forward_backward_accumulate_
void forward_backward_accumulate_(const int & chunk, int weight, int & expected_counts)Forward-backward to get expected piece counts for one chunk under the current probability model. Used by EM training. Adds to expected_counts (indexed by piece id).
EM training. See the file header for the algorithm; key knobs: * target_vocab_size — stop pruning when reached. * num_iterations — EM passes before each prune step. * shrink_factor — fraction of pieces removed per iteration (default 0.75 — matches SentencePiece "shrinking_factor"). * max_piece_length — cap on seed substring length. * initial_vocab_multiplier — seed vocab is sized at multiplier × target_vocab_size before pruning starts.
train_with_options
void train_with_options(const int & corpus, int target_vocab_size, int num_iterations, double shrink_factor, int max_piece_length, int initial_vocab_multiplier)Extended train with all knobs exposed.
Viterbi encode for one chunk.