![]() |
An behavioral cloning Agent.
Inherits From: TFAgent
tf_agents.agents.BehavioralCloningAgent(
time_step_spec: tf_agents.trajectories.time_step.TimeStep
,
action_spec: tf_agents.typing.types.NestedTensorSpec
,
cloning_network: tf_agents.networks.network.Network
,
optimizer: tf_agents.typing.types.Optimizer
,
num_outer_dims: int = 1,
epsilon_greedy: tf_agents.typing.types.Float
= 0.1,
loss_fn: tf_agents.typing.types.LossFn
= None,
gradient_clipping: Optional[types.Float] = None,
debug_summaries: bool = False,
summarize_grads_and_vars: bool = False,
train_step_counter: Optional[tf.Variable] = None,
name: Optional[Text] = None
)
Implements behavioral cloning, wherein the network learns to clone
given experience. Users must provide their own loss functions. Note this
implementation will use a QPolicy. To use with other policies subclass this
agent and override the _get_policies
method. Note the cloning_network must
match the requirements of the generated policies.
Behavioral cloning was proposed in the following articles:
Pomerleau, D.A., 1991. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1), pp.88-97.
Russell, S., 1998, July. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory (pp. 101-103). ACM.
Args | |
---|---|
time_step_spec
|
A TimeStep spec of the expected time_steps.
|
action_spec
|
A nest of BoundedTensorSpec representing the actions. |
cloning_network
|
A tf_agents.network.Network to be used by the agent.
The network will be called as
and must return a 2-tuple with elements |
optimizer
|
The optimizer to use for training. |
num_outer_dims
|
The number of outer dimensions for the agent. Must be either 1 or 2. If 2, training will require both a batch_size and time dimension on every Tensor; if 1, training will require only a batch_size outer dimension. |
epsilon_greedy
|
probability of choosing a random action in the default epsilon-greedy collect policy (used only if a wrapper is not provided to the collect_policy method). |
loss_fn
|
A function for computing the error between the output of the
cloning network and the action that was taken. If None, the loss
depends on the action dtype. If the dtype is integer, then loss_fn
is
If the dtype is floating point, the loss is
|
gradient_clipping
|
Norm length to clip gradients. |
debug_summaries
|
A bool to gather debug summaries. |
summarize_grads_and_vars
|
If True, gradient and network variable summaries will be written during training. |
train_step_counter
|
An optional counter to increment every time the train op is run. Defaults to the global_step. |
name
|
The name of this agent. All variables in this module will fall under that name. Defaults to the class name. |
Raises | |
---|---|
ValueError
|
If action_spec contains more than one action, but a custom
loss_fn is not provided.
|
Attributes | |
---|---|
action_spec
|
TensorSpec describing the action produced by the agent. |
collect_data_spec
|
Returns a Trajectory spec, as expected by the collect_policy .
|
collect_policy
|
Return a policy that can be used to collect data from the environment. |
data_context
|
|
debug_summaries
|
|
policy
|
Return the current policy held by the agent. |
summaries_enabled
|
|
summarize_grads_and_vars
|
|
time_step_spec
|
Describes the TimeStep tensors expected by the agent.
|
train_argspec
|
TensorSpec describing extra supported kwargs to train() .
|
train_sequence_length
|
The number of time steps needed in experience tensors passed to train .
Train requires experience to be a For example, for non-RNN DQN training, If this value is |
train_step_counter
|
|
training_data_spec
|
Returns a trajectory spec, as expected by the train() function. |
validate_args
|
Whether train & preprocess_sequence validate input & output args.
|
Methods
initialize
initialize() -> Optional[tf.Operation]
Initializes the agent.
Returns | |
---|---|
An operation that can be used to initialize the agent. |
Raises | |
---|---|
RuntimeError
|
If the class was not initialized properly (super.__init__
was not called).
|
preprocess_sequence
preprocess_sequence(
experience: tf_agents.typing.types.NestedTensor
) -> tf_agents.typing.types.NestedTensor
Defines preprocess_sequence function to be fed into replay buffers.
This defines how we preprocess the collected data before training.
Defaults to pass through for most agents.
Structure of experience
must match that of self.collect_data_spec
.
Args | |
---|---|
experience
|
a Trajectory shaped [batch, time, ...] or [time, ...] which
represents the collected experience data.
|
Returns | |
---|---|
A post processed Trajectory with the same shape as the input.
|
Raises | |
---|---|
TypeError
|
If experience does not match self.collect_data_spec structure
types.
|
train
train(
experience: tf_agents.typing.types.NestedTensor
,
weights: Optional[types.Tensor] = None,
**kwargs
) -> tf_agents.agents.tf_agent.LossInfo
Trains the agent.
Args | |
---|---|
experience
|
A batch of experience data in the form of a Trajectory . The
structure of experience must match that of self.training_data_spec .
All tensors in experience must be shaped [batch, time, ...] where
time must be equal to self.train_step_length if that
property is not None .
|
weights
|
(optional). A Tensor , either 0-D or shaped [batch] ,
containing weights to be used when calculating the total train loss.
Weights are typically multiplied elementwise against the per-batch loss,
but the implementation is up to the Agent.
|
**kwargs
|
Any additional data as declared by self.train_argspec .
|
Returns | |
---|---|
A LossInfo loss tuple containing loss and info tensors.
|
Raises | |
---|---|
TypeError
|
If validate_args is True and: Experience is not type
Trajectory ; or if experience does not match
self.training_data_spec structure types.
|
ValueError
|
If validate_args is True and: Experience tensors' time axes
are not compatible with self.train_sequence_length ; or if experience
does not match self.training_data_spec structure.
|
ValueError
|
If validate_args is True and the user does not pass
**kwargs matching self.train_argspec .
|
RuntimeError
|
If the class was not initialized properly (super.__init__
was not called).
|