A cache that grows its key/value tensors by concatenation each step.
Holds one key tensor and one value tensor per layer in key_cache and
value_cache (see the Attributes section). The
first update for a layer seeds the
entry; subsequent updates concatenate the new token(s) along the sequence
dimension. This is the eager default for autoregressive decoding.
Attributes
Examples
>>> import lucid
>>> from lucid.utils.cache import DynamicCache
>>> cache = DynamicCache()
>>> k = lucid.zeros(1, 2, 3, 4) # (B, H, T, D)
>>> v = lucid.zeros(1, 2, 3, 4)
>>> ck, cv = cache.update(k, v, layer_idx=0)
>>> cache.get_seq_length()
3Used by 2
Constructors
1Properties
1Class methods
1Instance methods
9Repeat every batch row repeats times along the batch dimension
(in place) — e.g. to fan one prompt out into repeats samples.
Keep only the batch rows named by indices along the batch
dimension (in place).
Truncate the cache to at most max_length tokens along the
sequence dimension (in place).
A negative max_length drops that many tokens off the end. Useful
for rolling the cache back (e.g. rejecting speculative tokens). A
max_length at or above the current length is a no-op.
Always None — the dynamic cache is unbounded and grows on demand.
Number of cached tokens for layer_idx (0 if not yet seeded),
read directly from the accumulated tensor's sequence dimension.
Reindex every layer's keys/values along the batch dimension with
beam_idx (in place) to keep beams aligned after a beam permutation.
Empty the cache in place so the same object can be reused for a fresh sequence (clears every layer and the token counter).
Convert to the legacy tuple(tuple(key, value), ...) format.
Returns one (key, value) pair per layer, in layer order — the
representation older decoder forward signatures consumed before
Cache objects existed.
update(key_states: Tensor, value_states: Tensor, layer_idx: int, cache_kwargs: dict[str, object] | None = None)Concatenate key_states / value_states onto layer_idx and
return the grown key/value tensors.
The first call for a layer seeds the entry; later calls
lucid.cat the new token(s) onto the sequence dimension.
Parameters
Returns
Dunder methods
3Return the (keys, values) pair cached for layer_idx.
Iterate (keys, values) pairs in layer order.
Number of layers currently held in the cache.