EncoderDecoderCache
CacheEncoderDecoderCache(self_attention_cache: _KVCache, cross_attention_cache: _KVCache, is_updated: dict[int, bool] | None = None)Wraps a decoder self-attention cache and a cross-attention cache.
Parameters
self_attention_cacheDynamicCache or StaticCachecross_attention_cacheDynamicCache or StaticCachememory then reused unchanged.is_updated(dict[int, bool] or None, optional, keyword - only)= NoneNone
(default) they are inferred from cross_attention_cache's sequence
length — correct for eager use. The compiled decode path passes them
explicitly (all True) because a re-wrapped StaticCache resets its
length counter, so length can no longer signal filled-ness.Attributes
self_attention_cacheDynamicCache or StaticCachecross_attention_cacheDynamicCache or StaticCacheis_updateddict[int, bool]memory projection on every step after the first.Used by 1
Constructors
1Class methods
1Instance methods
9Repeat every batch row repeats times in both sub-caches.
Keep only the batch rows named by indices in both sub-caches.
Truncate the decoder self-attention cache to max_length tokens.
The cross-attention cache is the fixed-length encoder memory and is left untouched.
Maximum decoder length, delegated to the self-attention sub-cache
(None if it is a growing DynamicCache).
Decoder sequence length for layer_idx (from the self-attention
sub-cache; the cross-attention length is fixed and not counted).
Reindex both sub-caches along the batch dimension with beam_idx
(in place) so self- and cross-attention stay beam-aligned.
Empty both sub-caches and clear the cross-attention 'updated' flags.
Convert to the legacy per-layer (self_k, self_v, cross_k, cross_v)
tuple format.
update(key_states: Tensor, value_states: Tensor, layer_idx: int, cache_kwargs: dict[str, object] | None = None)Update the decoder self-attention cache for layer_idx.
Cross-attention key/value are written directly through the
cross_attention_cache by the attention module (guarded by
is_updated), not through this method — so this delegates only to the
self-attention sub-cache.
Parameters
key_statesTensor(B, num_heads, T_new, head_dim).value_statesTensorlayer_idxintcache_kwargsdict or None= NoneStaticCache
reads "cache_position" from it).Returns
Dunder methods
3Return (self_k, self_v, cross_k, cross_v) for layer_idx.
Iterate (self_k, self_v, cross_k, cross_v) tuples in layer order.
Number of decoder layers (from the self-attention cache).