tf_agents.agents.ddpg.actor_rnn_network.ActorRnnNetwork

View source on GitHub

Creates a recurrent actor network.

Inherits From: Network

input_tensor_spec A nest of tensor_spec.TensorSpec representing the input observations.
output_tensor_spec A nest of tensor_spec.BoundedTensorSpec representing the actions.
conv_layer_params Optional list of convolution layers parameters, where each item is a length-three tuple indicating (filters, kernel_size, stride).
input_fc_layer_params Optional list of fully_connected parameters, where each item is the number of units in the layer. This is applied before the LSTM cell.
lstm_size An iterable of ints specifying the LSTM cell sizes to use.
output_fc_layer_params Optional list of fully_connected parameters, where each item is the number of units in the layer. This is applied after the LSTM cell.
activation_fn Activation function, e.g. tf.nn.relu, slim.leaky_relu, ...
name A string representing name of the network.

ValueError If input_tensor_spec contains more than one observation.

activity_regularizer Optional regularizer function for the output of this layer.
dtype Dtype used by the weights of the layer, set in the constructor.
dynamic Whether the layer is dynamic (eager-only); set in the constructor.
input Retrieves the input tensor(s) of a layer.

Only applicable if the layer has exactly one input, i.e. if it is connected to one incoming layer.

input_spec Gets the network's input specs.
input_tensor_spec Returns the spec of the input to the network of type InputSpec.
layers

losses Losses which are associated with this Layer.

Variable regularization tensors are created when this property is accessed, so it is eager safe: accessing losses under a tf.GradientTape will propagate gradients back to the corresponding variables.

metrics List of tf.keras.metrics.Metric instances tracked by the layer.
name Name of the layer (string), set in the constructor.
name_scope Returns a tf.name_scope instance for this class.
non_trainable_weights List of all non-trainable weights tracked by this layer.

Non-trainable weights are not updated during training. They are expected to be updated manually in call().

output Retrieves the output tensor(s) of a layer.

Only applicable if the layer has exactly one output, i.e. if it is connected to one incoming layer.

state_spec

state_updates Returns the updates from all layers that are stateful.

This is useful for separating training updates and state updates, e.g. when we need to update a layer's internal state during prediction.

submodules Sequence of all sub-modules.

Submodules are modules which are properties of this module, or found as properties of modules which are properties of this module (and so on).

a = tf.Module()
b = tf.Module()
c = tf.Module()
a.b = b
b.c = c
list(a.submodules) == [b, c]
True
list(b.submodules) == [c]
True
list(c.submodules) == []
True

trainable

trainable_variables Sequence of trainable variables owned by this module and its submodules.

trainable_weights List of all trainable weights tracked by this layer.

Trainable weights are updated via gradient descent during training.

variables Returns the list of all layer variables/weights.

Alias of self.weights.

weights Returns the list of all layer variables/weights.

Methods

add_loss

Add loss tensor(s), potentially dependent on layer inputs.

Some losses (for instance, activity regularization losses) may be dependent on the inputs passed when calling a layer. Hence, when reusing the same layer on different inputs a and b, some entries in layer.losses may be dependent on a and some on b. This method automatically keeps track of dependencies.

This method can be used inside a subclassed layer or model's call function, in which case losses should be a Tensor or list of Tensors.

Example:

class MyLayer(tf.keras.layers.Layer):
  def call(inputs, self):
    self.add_loss(tf.abs(tf.reduce_mean(inputs)), inputs=True)
    return inputs

This method can also be called directly on a Functional Model during construction. In this case, any loss Tensors passed to this Model must be symbolic and be able to be traced back to the model's Inputs. These losses become part of the model's topology and are tracked in get_config.

Example:

inputs = tf.keras.Input(shape=(10,))
x = tf.keras.layers.Dense(10)(inputs)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
# Activity regularization.
model.add_loss(tf.abs(tf.reduce_mean(x)))

If this is not the case for your loss (if, for example, your loss references a Variable of one of the model's layers), you can wrap your loss in a zero-argument lambda. These losses are not tracked as part of the model's topology since they can't be serialized.

Example:

inputs = tf.keras.Input(shape=(10,))
x = tf.keras.layers.Dense(10)(inputs)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
# Weight regularization.
model.add_loss(lambda: tf.reduce_mean(x.kernel))

The get_losses_for method allows to retrieve the losses relevant to a specific set of inputs.

Arguments
losses Loss tensor, or list/tuple of tensors. Rather than tensors, losses may also be zero-argument callables which create a loss tensor.
inputs Ignored when executing eagerly. If anything other than None is passed, it signals the losses are conditional on some of the layer's inputs, and thus they should only be run where these inputs are available. This is the case for activity regularization losses, for instance. If None is passed, the losses are assumed to be unconditional, and will apply across all dataflows of the layer (e.g. weight regularization losses).

add_metric

Adds metric tensor to the layer.

Args
value Metric tensor.
aggregation Sample-wise metric reduction function. If aggregation=None, it indicates that the metric tensor provided has been aggregated already. eg, bin_acc = BinaryAccuracy(name='acc') followed by model.add_metric(bin_acc(y_true, y_pred)). If aggregation='mean', the given metric tensor will be sample-wise reduced using mean function. eg, model.add_metric(tf.reduce_sum(outputs), name='output_mean', aggregation='mean').
name String metric name.

Raises
ValueError If aggregation is anything other than None or mean.

build

Builds the model based on input shapes received.

This is to be used for subclassed models, which do not know at instantiation time what their inputs look like.

This method only exists for users who want to call model.build() in a standalone way (as a substitute for calling the model on real data to build it). It will never be called by the framework (and thus it will never throw unexpected errors in an unrelated workflow).

Args
input_shape Single tuple, TensorShape, or list of shapes, where shapes are tuples, integers, or TensorShapes.

Raises
ValueError

  1. In case of invalid user-provided data (not of type tuple, list, or TensorShape).
  2. If the model requires call arguments that are agnostic to the input shapes (positional or kwarg in call signature).
  3. If not all layers were properly built.
  4. If float type inputs are not supported within the layers.

In each of these cases, the user should build their model by calling it on real tensor data.

compute_mask

Computes an output mask tensor.

Arguments
inputs Tensor or list of tensors.
mask Tensor or list of tensors.

Returns
None or a tensor (or list of tensors, one per output tensor of the layer).

compute_output_shape

Computes the output shape of the layer.

If the layer has not been built, this method will call build on the layer. This assumes that the layer will later be used with inputs that match the input shape provided here.

Arguments
input_shape Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.

Returns
An input shape tuple.

copy

View source

Create a shallow copy of this network.

Args
**kwargs Args to override when recreating this network. Commonly overridden args include 'name'.

Returns
A shallow copy of this network.

count_params

Count the total number of scalars composing the weights.

Returns
An integer count.

Raises
ValueError if the layer isn't yet built (in which case its weights aren't yet defined).

create_variables

View source

from_config

Instantiates a Model from its config (output of get_config()).

Arguments
config Model config dictionary.
custom_objects Optional dictionary mapping names (strings) to custom classes or functions to be considered during deserialization.

Returns
A model instance.

Raises
ValueError In case of improperly formatted config dict.

get_config

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns
Python dictionary.

get_initial_state

View source

Returns an initial state usable by the network.

Args
batch_size Tensor or constant: size of the batch dimension. Can be None in which case not dimensions gets added.

Returns
A nested object of type self.state_spec containing properly initialized Tensors.

get_layer

Retrieves a layer based on either its name (unique) or index.

If name and index are both provided, index will take precedence. Indices are based on order of horizontal graph traversal (bottom-up).

Arguments
name String, name of layer.
index Integer, index of layer.

Returns
A layer instance.

Raises
ValueError In case of invalid layer name or index.

get_weights

Returns the current weights of the layer.

The weights of a layer represent the state of the layer. This function returns both trainable and non-trainable weight values associated with this layer as a list of Numpy arrays, which can in turn be used to load state into similarly parameterized layers.

For example, a Dense layer returns a list of two values-- per-output weights and the bias value. These can be used to set the weights of another Dense layer:

a = tf.keras.layers.Dense(1,
  kernel_initializer=tf.constant_initializer(1.))
a_out = a(tf.convert_to_tensor([[1., 2., 3.]]))
a.get_weights()
[array([[1.],
       [1.],
       [1.]], dtype=float32), array([0.], dtype=float32)]
b = tf.keras.layers.Dense(1,
  kernel_initializer=tf.constant_initializer(2.))
b_out = b(tf.convert_to_tensor([[10., 20., 30.]]))
b.get_weights()
[array([[2.],
       [2.],
       [2.]], dtype=float32), array([0.], dtype=float32)]
b.set_weights(a.get_weights())
b.get_weights()
[array([[1.],
       [1.],
       [1.]], dtype=float32), array([0.], dtype=float32)]

Returns
Weights values as a list of numpy arrays.

load_weights

Loads all layer weights, either from a TensorFlow or an HDF5 weight file.

If by_name is False weights are loaded based on the network's topology. This means the architecture should be the same as when the weights were saved. Note that layers that don't have weights are not taken into account in the topological ordering, so adding or removing layers is fine as long as they don't have weights.

If by_name is True, weights are loaded into layers only if they share the same name. This is useful for fine-tuning or transfer-learning models where some of the layers have changed.

Only topological loading (by_name=False) is supported when loading weights from the TensorFlow format. Note that topological loading differs slightly between TensorFlow and HDF5 formats for user-defined classes inheriting from tf.keras.Model: HDF5 loads based on a flattened list of weights, while the TensorFlow format loads based on the object-local names of attributes to which layers are assigned in the Model's constructor.

Arguments
filepath String, path to the weights file to load. For weight files in TensorFlow format, this is the file prefix (the same as was passed to save_weights).
by_name Boolean, whether to load weights by name or by topological order. Only topological loading is supported for weight files in TensorFlow format.
skip_mismatch Boolean, whether to skip loading of layers where there is a mismatch in the number of weights, or a mismatch in the shape of the weight (only valid when by_name=True).

Returns
When loading a weight file in TensorFlow format, returns the same status object as tf.train.Checkpoint.restore. When graph building, restore ops are run automatically as soon as the network is built (on first call for user-defined classes inheriting from Model, immediately if it is already built).

When loading weights in HDF5 format, returns None.

Raises
ImportError If h5py is not available and the weight file is in HDF5 format.
ValueError If skip_mismatch is set to True when by_name is False.

reset_states

save

Saves the model to Tensorflow SavedModel or a single HDF5 file.

The savefile includes:

  • The model architecture, allowing to re-instantiate the model.
  • The model weights.
  • The state of the optimizer, allowing to resume training exactly where you left off.

This allows you to save the entirety of the state of a model in a single file.

Saved models can be reinstantiated via keras.models.load_model. The model returned by load_model is a compiled model ready to be used (unless the saved model was never compiled in the first place).

Models built with the Sequential and Functional API can be saved to both the HDF5 and SavedModel formats. Subclassed models can only be saved with the SavedModel format.

Note that the model weights may have different scoped names after being loaded. Scoped names include the model/layer names, such as "dense_1/kernel:0". It is recommended that you use the layer properties to access specific variables, e.g.model.get_layer("dense_1").kernel`.

Arguments
filepath String, path to SavedModel or H5 file to save the model.
overwrite Whether to silently overwrite any existing file at the target location, or provide the user with a manual prompt.
include_optimizer If True, save optimizer's state together.
save_format Either 'tf' or 'h5', indicating whether to save the model to Tensorflow SavedModel or HDF5. Defaults to 'tf' in TF 2.X, and 'h5' in TF 1.X.
signatures Signatures to save with the SavedModel. Applicable to the 'tf' format only. Please see the signatures argument in tf.saved_model.save for details.
options Optional tf.saved_model.SaveOptions object that specifies options for saving to SavedModel.

Example:

from keras.models import load_model

model.save('my_model.h5')  # creates a HDF5 file 'my_model.h5'
del model  # deletes the existing model

# returns a compiled model
# identical to the previous one
model = load_model('my_model.h5')

save_weights

Saves all layer weights.

Either saves in HDF5 or in TensorFlow format based on the save_format argument.

When saving in HDF5 format, the weight file has:

  • layer_names (attribute), a list of strings (ordered names of model layers).
  • For every layer, a group named layer.name
    • For every such layer group, a group attribute weight_names, a list of strings (ordered names of weights tensor of the layer).
    • For every weight in the layer, a dataset storing the weight value, named after the weight tensor.

When saving in TensorFlow format, all objects referenced by the network are saved in the same format as tf.train.Checkpoint, including any Layer instances or Optimizer instances assigned to object attributes. For networks constructed from inputs and outputs using tf.keras.Model(inputs, outputs), Layer instances used by the network are tracked/saved automatically. For user-defined classes which inherit from tf.keras.Model, Layer instances must be assigned to object attributes, typically in the constructor. See the documentation of tf.train.Checkpoint and tf.keras.Model for details.

While the formats are the same, do not mix save_weights and tf.train.Checkpoint. Checkpoints saved by Model.save_weights should be loaded using Model.load_weights. Checkpoints saved using tf.train.Checkpoint.save should be restored using the corresponding tf.train.Checkpoint.restore. Prefer tf.train.Checkpoint over save_weights for training checkpoints.

The TensorFlow format matches objects and variables by starting at a root object, self for save_weights, and greedily matching attribute names. For Model.save this is the Model, and for Checkpoint.save this is the Checkpoint even if the Checkpoint has a model attached. This means saving a tf.keras.Model using save_weights and loading into a tf.train.Checkpoint with a Model attached (or vice versa) will not match the Model's variables. See the guide to training checkpoints for details on the TensorFlow format.

Arguments
filepath String, path to the file to save the weights to. When saving in TensorFlow format, this is the prefix used for checkpoint files (multiple files are generated). Note that the '.h5' suffix causes weights to be saved in HDF5 format.
overwrite Whether to silently overwrite any existing file at the target location, or provide the user with a manual prompt.
save_format Either 'tf' or 'h5'. A filepath ending in '.h5' or '.keras' will default to HDF5 if save_format is None. Otherwise None defaults to 'tf'.

Raises
ImportError If h5py is not available when attempting to save in HDF5 format.
ValueError For invalid/unknown format arguments.

set_weights

Sets the weights of the layer, from Numpy arrays.

The weights of a layer represent the state of the layer. This function sets the weight values from numpy arrays. The weight values should be passed in the order they are created by the layer. Note that the layer's weights must be instantiated before calling this function by calling the layer.

For example, a Dense layer returns a list of two values-- per-output weights and the bias value. These can be used to set the weights of another Dense layer:

a = tf.keras.layers.Dense(1,
  kernel_initializer=tf.constant_initializer(1.))
a_out = a(tf.convert_to_tensor([[1., 2., 3.]]))
a.get_weights()
[array([[1.],
       [1.],
       [1.]], dtype=float32), array([0.], dtype=float32)]
b = tf.keras.layers.Dense(1,
  kernel_initializer=tf.constant_initializer(2.))
b_out = b(tf.convert_to_tensor([[10., 20., 30.]]))
b.get_weights()
[array([[2.],
       [2.],
       [2.]], dtype=float32), array([0.], dtype=float32)]
b.set_weights(a.get_weights())
b.get_weights()
[array([[1.],
       [1.],
       [1.]], dtype=float32), array([0.], dtype=float32)]

Arguments
weights a list of Numpy arrays. The number of arrays and their shape must match number of the dimensions of the weights of the layer (i.e. it should match the output of get_weights).

Raises
ValueError If the provided weights list does not match the layer's specifications.

summary

Prints a string summary of the network.

Arguments
line_length Total length of printed lines (e.g. set this to adapt the display to different terminal window sizes).
positions Relative or absolute positions of log elements in each line. If not provided, defaults to [.33, .55, .67, 1.].
print_fn Print function to use. Defaults to print. It will be called on each line of the summary. You can set it to a custom function in order to capture the string summary.

Raises
ValueError if summary() is called before the model is built.

to_json

Returns a JSON string containing the network configuration.

To load a network from a JSON save file, use keras.models.model_from_json(json_string, custom_objects={}).

Arguments
**kwargs Additional keyword arguments to be passed to json.dumps().

Returns
A JSON string.

to_yaml

Returns a yaml string containing the network configuration.

To load a network from a yaml save file, use keras.models.model_from_yaml(yaml_string, custom_objects={}).

custom_objects should be a dictionary mapping the names of custom losses / layers / etc to the corresponding functions / classes.

Arguments
**kwargs Additional keyword arguments to be passed to yaml.dump().

Returns
A YAML string.

Raises
ImportError if yaml module is not found.

with_name_scope

Decorator to automatically enter the module name scope.

class MyModule(tf.Module):
  @tf.Module.with_name_scope
  def __call__(self, x):
    if not hasattr(self, 'w'):
      self.w = tf.Variable(tf.random.normal([x.shape[1], 3]))
    return tf.matmul(x, self.w)

Using the above module would produce tf.Variables and tf.Tensors whose names included the module name:

mod = MyModule()
mod(tf.ones([1, 2]))
<tf.Tensor: shape=(1, 3), dtype=float32, numpy=..., dtype=float32)>
mod.w
<tf.Variable 'my_module/Variable:0' shape=(2, 3) dtype=float32,
numpy=..., dtype=float32)>

Args
method The method to wrap.

Returns
The original method wrapped such that it enters the module's name scope.

__call__

View source

A wrapper around Network.call.

A typical call method in a class subclassing Network looks like this:

def call(self,
         observation,
         step_type=None,
         network_state=(),
         training=False):
    ...
    return outputs, new_network_state

In this case, we will validate the first argument (observation) against self.input_tensor_spec.

If a network_state kwarg is given it is also validated against self.state_spec. Similarly, the return value of the call method is expected to be a tuple/list with 2 values: (output, new_state); we validate new_state against self.state_spec.

Args
inputs The inputs to self.call, matching self.input_state_spec.
*args Additional arguments to self.call.
**kwargs Additional keyword arguments to self.call.

Returns
A tuple (outputs, new_network_state).