data

DreamerConfig

extendsWorldModelConfig

DreamerConfig(sample_size: int | tuple[int, int] = WORLD_MODEL_IMAGE_SIZE, in_channels: int = 3, out_channels: int = 3, act_fn: GenerativeActivation = 'elu', action_dim: int = 1, stoch_size: int = 30, deter_size: int = 200, hidden_size: int = 200, cnn_depth: int = 32, min_std: float = 0.1, mean_only: bool = False, free_nats: float = 3.0, kl_weight: float = 1.0, cnn_act: GenerativeActivation = 'relu', horizon: int = 15, discount: float = 0.99, lambda_: float = 0.95, actor_hidden: int = 300, actor_layers: int = 3, value_hidden: int = 300, value_layers: int = 3, actor_min_std: float = 0.0001, actor_init_std: float = 5.0, actor_mean_scale: float = 5.0, reward_hidden: int = 300, reward_layers: int = 3, detach_actor_input: bool = True, pcont: bool = False, pcont_scale: float = 10.0, pcont_layers: int = 3, action_space: str = 'continuous')

source edit

Frozen configuration for the Dreamer family.

Defaults reproduce Hafner et al., 2020 on the DeepMind Control Suite. Fields shared with PlaNet are inherited from WorldModelConfig.

Parameters

act_fn(silu, swish, relu, gelu, elu)= "silu"

Activation throughout. The paper specifies ELU, where PlaNet specifies ReLU — the two families genuinely differ here.

cnn_act(silu, swish, relu, gelu, elu)= "silu"

Activation for the convolutional encoder and decoder only. The released implementation carries two — dense_act is ELU and cnn_act is ReLU — and the paper states the first while saying nothing about the second, so this follows the rule the rest of this class uses when the paper is silent.

horizonint= 15

How many steps each imagined trajectory runs for. Short on purpose: model error compounds, and the critic's bootstrap is what carries value beyond the horizon.

discountfloat= 0.99

Reward discount

\gamma

lambda_float= 0.95

TD(

\lambda

) weighting between short- and long-horizon returns. Trailing underscore because lambda is a keyword.

actor_hiddenint= 300, 3

Width and depth of the actor.

actor_layersint= 300, 3

Width and depth of the actor.

value_hiddenint= 300, 3

Width and depth of the critic.

value_layersint= 300, 3

Width and depth of the critic.

actor_min_stdfloat= 1e-4

Floor on the action distribution's scale.

actor_init_stdfloat= 5.0

Offset added before softplus so the policy starts wide enough to explore.

actor_mean_scalefloat= 5.0

The mean is passed through s * tanh(x / s) before squashing, which keeps it from saturating the outer tanh.

reward_hiddenint= 300, 3

Reward head. Appendix A gives one shape for everything it does not name separately — "all other functions as three dense layers of size 300" — so the reward, action and value models share it. The released implementation disagrees on both counts (400 units, with two layers for reward and four for the actor); the paper is followed here, as everywhere else in this family. Width and depth of the reward head. The paper names only the action and value models, so the depth here is the released implementation's — which also matches PlaNet's reward head.

reward_layersint= 300, 3

detach_actor_inputbool= True

Hide the state's gradient from the actor during imagination. The released implementation does this; the paper does not discuss it. Setting False gives the exact gradient of the stated objective — see Notes.

pcontbool= False

Predict the discount factor from each latent state instead of holding it constant. The paper introduces this for "tasks with early termination"; with no termination there is nothing to learn, which is why it is off by default.

pcont_scalefloat= 10.0

Multiplier on the discount head's likelihood.

pcont_layersint= 3

Depth of the discount head. Width follows value_hidden.

action_space(continuous, discrete)= "continuous"

What an action is. "continuous" is the tanh-squashed Gaussian the Control Suite needs. "discrete" makes the action model "predict the logits of a categorical distribution", which is what the paper's Discrete control paragraph specifies for Atari and DeepMind Lab, sampled with straight-through gradients so the actor's gradient still arrives through the action.

Notes

Reference: Hafner, Lillicrap, Ba, and Norouzi, "Dream to Control: Learning Behaviors by Latent Imagination", ICLR, 2020 (arXiv:1912.01603).

The actor emits a tanh-squashed diagonal Gaussian, so actions are bounded to (-1, 1) — the Control Suite's range. actor_min_std, actor_init_std and actor_mean_scale are not stated in the paper; they are the released implementation's, and are exposed here rather than buried so a reader can see which numbers have a citation and which have a source.

Where the paper and the released implementation disagree, the paper wins and the difference is recorded here rather than hidden:

============== ==================== ========================== quantity paper released implementation ============== ==================== ========================== head width 300 400 throughout actor depth 3 4 value depth 3 3 reward depth not stated 2 (taken, as the default) CNN activation not stated ReLU (taken; dense stays ELU) ============== ==================== ==========================

detach_actor_input is the one behavioural fork. The released implementation feeds the actor a stop_gradient-ed state during imagination, which drops the terms in which a return depends on the policy through the state it read. Keeping them (False) is the exact gradient of $\max_\phi \mathbb{E}[\sum_\tau V_\lambda]$ ; dropping them is cheaper and lower-variance. The default follows the released implementation because the paper is silent.

The three losses need three optimisers over three parameter groups — see DreamerForWorldModeling. Summing them and taking one step is not this algorithm: the actor's gradient would then also descend the world model, and the critic would chase a target it is simultaneously moving.

Examples

>>> from lucid.models.generative.dreamer import DreamerConfig
>>> cfg = DreamerConfig(action_dim=6)
>>> cfg.horizon, cfg.lambda_, cfg.discount
(15, 0.95, 0.99)
>>> cfg.act_fn
'elu'

Used by 3

Constructors

dunder

init

→None

__init__(sample_size: int | tuple[int, int] = WORLD_MODEL_IMAGE_SIZE, in_channels: int = 3, out_channels: int = 3, act_fn: GenerativeActivation = 'elu', action_dim: int = 1, stoch_size: int = 30, deter_size: int = 200, hidden_size: int = 200, cnn_depth: int = 32, min_std: float = 0.1, mean_only: bool = False, free_nats: float = 3.0, kl_weight: float = 1.0, cnn_act: GenerativeActivation = 'relu', horizon: int = 15, discount: float = 0.99, lambda_: float = 0.95, actor_hidden: int = 300, actor_layers: int = 3, value_hidden: int = 300, value_layers: int = 3, actor_min_std: float = 0.0001, actor_init_std: float = 5.0, actor_mean_scale: float = 5.0, reward_hidden: int = 300, reward_layers: int = 3, detach_actor_input: bool = True, pcont: bool = False, pcont_scale: float = 10.0, pcont_layers: int = 3, action_space: str = 'continuous')

source edit

Initialise the actor. See the class docstring for parameters.

DreamerConfig(sample_size: int | tuple[int, int] = WORLD_MODEL_IMAGE_SIZE, in_channels: int = 3, out_channels: int = 3, act_fn: GenerativeActivation = 'elu', action_dim: int = 1, stoch_size: int = 30, deter_size: int = 200, hidden_size: int = 200, cnn_depth: int = 32, min_std: float = 0.1, mean_only: bool = False, free_nats: float = 3.0, kl_weight: float = 1.0, cnn_act: GenerativeActivation = 'relu', horizon: int = 15, discount: float = 0.99, lambda_: float = 0.95, actor_hidden: int = 300, actor_layers: int = 3, value_hidden: int = 300, value_layers: int = 3, actor_min_std: float = 0.0001, actor_init_std: float = 5.0, actor_mean_scale: float = 5.0, reward_hidden: int = 300, reward_layers: int = 3, detach_actor_input: bool = True, pcont: bool = False, pcont_scale: float = 10.0, pcont_layers: int = 3, action_space: str = 'continuous')

__init__(sample_size: int | tuple[int, int] = WORLD_MODEL_IMAGE_SIZE, in_channels: int = 3, out_channels: int = 3, act_fn: GenerativeActivation = 'elu', action_dim: int = 1, stoch_size: int = 30, deter_size: int = 200, hidden_size: int = 200, cnn_depth: int = 32, min_std: float = 0.1, mean_only: bool = False, free_nats: float = 3.0, kl_weight: float = 1.0, cnn_act: GenerativeActivation = 'relu', horizon: int = 15, discount: float = 0.99, lambda_: float = 0.95, actor_hidden: int = 300, actor_layers: int = 3, value_hidden: int = 300, value_layers: int = 3, actor_min_std: float = 0.0001, actor_init_std: float = 5.0, actor_mean_scale: float = 5.0, reward_hidden: int = 300, reward_layers: int = 3, detach_actor_input: bool = True, pcont: bool = False, pcont_scale: float = 10.0, pcont_layers: int = 3, action_space: str = 'continuous')