class

PrioritizedSequenceReplay

extendsSequenceReplay

PrioritizedSequenceReplay(capacity: int = 1000000, alpha: float = 0.6, beta: float = 0.4, epsilon: float = 1e-06)

source edit

Sequence replay that samples in proportion to a priority.

Parameters

capacityint= 1_000_000

As SequenceReplay, in transitions.

alphafloat= 0.6

How much prioritization is used. 0 is uniform.

betafloat= 0.4

Starting exponent for the importance-sampling correction. The paper anneals it to 1 over training; see anneal_beta.

epsilonfloat= 1e-6

Added to every priority so that a chunk whose error reached zero can still be drawn again.

Raises

ValueError

If alpha or beta is negative, or epsilon is not positive.

Notes

Reference: Schaul, Quan, Antonoglou, and Silver, "Prioritized Experience Replay", ICLR, 2016 (arXiv:1511.05952).

P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha}, \qquad w_i = \frac{(N \cdot P(i))^{-\beta}}{\max_j w_j},

with proportional priorities $p_i = |\delta_i| + \epsilon$ .

Prioritizes episodes, not individual transitions, because that is the unit this buffer stores and the unit a sequence model trains on. A chunk's priority is its episode's.

Two details from the paper are easy to leave out and both matter. A newly stored episode is given the largest priority seen so far, so that everything is replayed at least once before prioritization can bury it — Algorithm 1, line 6. And the weights are normalised by their maximum "so that they only scale the update downwards", which keeps the correction from inflating any gradient.

Examples

>>> from lucid.utils.rollout import PrioritizedSequenceReplay
>>> buffer = PrioritizedSequenceReplay(capacity=1000)
>>> buffer.alpha, buffer.beta
(0.6, 0.4)

Used by 1

lucid.utils.rollout

Constructors

dunder

init

→None

__init__(capacity: int = 1000000, alpha: float = 0.6, beta: float = 0.4, epsilon: float = 1e-06)

source edit

Initialise the buffer. See the class docstring for parameters.

Instance methods

add

→None

add(episode: Episode)

source edit

Store an episode at the largest priority seen.

Parameters

episodeEpisode

As SequenceReplay.add.

Notes

Eviction can drop episodes off the front, so the priorities are re-laid after the base class has done its bookkeeping rather than tracked incrementally — the buffer is a list and the indices move.

anneal_beta

→None

anneal_beta(fraction: float, final: float = 1.0)

source edit

Move beta linearly toward final.

Parameters

fractionfloat

Training progress in [0, 1].

finalfloat= 1.0

Where beta ends up, which the paper reaches "only at the end of learning".

Notes

The correction matters most near convergence, so it is annealed in rather than applied at full strength from the start — the paper's argument is that early training is non-stationary anyway.

sample_prioritized

→PrioritizedBatch

sample_prioritized(batch_size: int, length: int)

source edit

Draw chunks in proportion to priority, with their weights.

Parameters

batch_sizeint

As SequenceReplay.sample.

lengthint

As SequenceReplay.sample.

Returns

PrioritizedBatch

The chunks, where they came from, and the correction weights.

Raises

ValueError

If no stored episode is long enough, or nothing is stored.

update_priorities

→None

update_priorities(indices: list[int], errors: list[float])

source edit

Rewrite the priorities of the episodes just trained on.

Parameters

indiceslist of int

As returned by sample_prioritized.

errorslist of float

One magnitude per index — the paper uses |delta|, the temporal-difference error.

Raises

ValueError

If the two lists differ in length.

class

PrioritizedSequenceReplay

extendsSequenceReplay

PrioritizedSequenceReplay(capacity: int = 1000000, alpha: float = 0.6, beta: float = 0.4, epsilon: float = 1e-06)

source edit

Sequence replay that samples in proportion to a priority.

Parameters

capacityint= 1_000_000

As SequenceReplay, in transitions.

alphafloat= 0.6

How much prioritization is used. 0 is uniform.

betafloat= 0.4

Starting exponent for the importance-sampling correction. The paper anneals it to 1 over training; see anneal_beta.

epsilonfloat= 1e-6

Added to every priority so that a chunk whose error reached zero can still be drawn again.

Raises

ValueError

If alpha or beta is negative, or epsilon is not positive.

Notes

Reference: Schaul, Quan, Antonoglou, and Silver, "Prioritized Experience Replay", ICLR, 2016 (arXiv:1511.05952).

P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha}, \qquad w_i = \frac{(N \cdot P(i))^{-\beta}}{\max_j w_j},

with proportional priorities $p_i = |\delta_i| + \epsilon$ .

Prioritizes episodes, not individual transitions, because that is the unit this buffer stores and the unit a sequence model trains on. A chunk's priority is its episode's.

Examples

>>> from lucid.utils.rollout import PrioritizedSequenceReplay
>>> buffer = PrioritizedSequenceReplay(capacity=1000)
>>> buffer.alpha, buffer.beta
(0.6, 0.4)

Used by 1

lucid.utils.rollout

Constructors

dunder

init

→None

__init__(capacity: int = 1000000, alpha: float = 0.6, beta: float = 0.4, epsilon: float = 1e-06)

source edit

Initialise the buffer. See the class docstring for parameters.

Instance methods

add

→None

add(episode: Episode)

source edit

Store an episode at the largest priority seen.

Parameters

episodeEpisode

As SequenceReplay.add.

Notes

Eviction can drop episodes off the front, so the priorities are re-laid after the base class has done its bookkeeping rather than tracked incrementally — the buffer is a list and the indices move.

anneal_beta

→None

anneal_beta(fraction: float, final: float = 1.0)

source edit

Move beta linearly toward final.

Parameters

fractionfloat

Training progress in [0, 1].

finalfloat= 1.0

Where beta ends up, which the paper reaches "only at the end of learning".

Notes

The correction matters most near convergence, so it is annealed in rather than applied at full strength from the start — the paper's argument is that early training is non-stationary anyway.

sample_prioritized

→PrioritizedBatch

sample_prioritized(batch_size: int, length: int)

source edit

Draw chunks in proportion to priority, with their weights.

Parameters

batch_sizeint

As SequenceReplay.sample.

lengthint

As SequenceReplay.sample.

Returns

PrioritizedBatch

The chunks, where they came from, and the correction weights.

Raises

ValueError

If no stored episode is long enough, or nothing is stored.

update_priorities

→None

update_priorities(indices: list[int], errors: list[float])

source edit

Rewrite the priorities of the episodes just trained on.

Parameters

indiceslist of int

As returned by sample_prioritized.

errorslist of float

One magnitude per index — the paper uses |delta|, the temporal-difference error.

Raises

ValueError

If the two lists differ in length.