此页面由 Cloud Translation API 翻译。
Switch to English

SAC minitaur

版权所有2018 TF-Agents作者。

在TensorFlow.org上查看 在Google Colab中运行 在GitHub上查看源代码 下载笔记本

介绍

此示例说明如何使用TF-Agents库在Minitaur环境上训练“ 软角色评论家”代理。

如果您已经通过DQN Colab进行了工作,那应该会感到非常熟悉。显着的变化包括:

  • 将代理从DQN更改为SAC。
  • 在Minitaur上进行培训,该环境比CartPole复杂得多。 Minitaur环境旨在训练四足机器人前进。
  • 我们不使用随机策略执行初始数据收集。

如果尚未安装以下依赖项,请运行:

sudo apt-get install -y xvfb ffmpeg
pip install -q 'gym==0.10.11'
pip install -q 'imageio==2.4.0'
pip install -q matplotlib
pip install -q PILLOW
pip install -q --pre tf-agents[reverb]
pip install -q 'pybullet==2.4.2'



ffmpeg is already the newest version (7:3.4.8-0ubuntu0.2).
xvfb is already the newest version (2:1.19.6-1ubuntu4.4).
The following packages were automatically installed and are no longer required:
  dconf-gsettings-backend dconf-service dkms freeglut3 freeglut3-dev
  glib-networking glib-networking-common glib-networking-services
  gsettings-desktop-schemas libcairo-gobject2 libcolord2 libdconf1
  libegl1-mesa libepoxy0 libglu1-mesa libglu1-mesa-dev libgtk-3-0
  libgtk-3-common libice-dev libjansson4 libjson-glib-1.0-0
  libjson-glib-1.0-common libproxy1v5 librest-0.7-0 libsm-dev
  libsoup-gnome2.4-1 libsoup2.4-1 libxi-dev libxmu-dev libxmu-headers
  libxnvctrl0 libxt-dev linux-gcp-headers-5.0.0-1026
  linux-headers-5.0.0-1026-gcp linux-image-5.0.0-1026-gcp
  linux-modules-5.0.0-1026-gcp pkg-config policykit-1-gnome python3-xkit
  screen-resolution-extra xserver-xorg-core-hwe-18.04
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 90 not upgraded.
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/tmpfs/src/tf_docs_env/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/tmpfs/src/tf_docs_env/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/tmpfs/src/tf_docs_env/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/tmpfs/src/tf_docs_env/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/tmpfs/src/tf_docs_env/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.1.1; however, version 20.2 is available.
You should consider upgrading via the '/tmpfs/src/tf_docs_env/bin/python -m pip install --upgrade pip' command.

建立

首先,我们将导入所需的各种工具,并确保启用TF-V2行为,因为在整个Colab中更容易在Eager模式下进行迭代。

 from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import base64
import imageio
import IPython
import matplotlib
import matplotlib.pyplot as plt
import PIL.Image

import tensorflow as tf
tf.compat.v1.enable_v2_behavior()

from tf_agents.agents.ddpg import critic_network
from tf_agents.agents.sac import sac_agent
from tf_agents.drivers import dynamic_step_driver
from tf_agents.environments import suite_pybullet
from tf_agents.environments import tf_py_environment
from tf_agents.eval import metric_utils
from tf_agents.metrics import tf_metrics
from tf_agents.networks import actor_distribution_network
from tf_agents.networks import normal_projection_network
from tf_agents.policies import greedy_policy
from tf_agents.policies import random_tf_policy
from tf_agents.replay_buffers import tf_uniform_replay_buffer
from tf_agents.trajectories import trajectory
from tf_agents.utils import common

 

超参数

 env_name = "MinitaurBulletEnv-v0" # @param {type:"string"}

# use "num_iterations = 1e6" for better results,
# 1e5 is just so this doesn't take too long. 
num_iterations = 100000 # @param {type:"integer"}

initial_collect_steps = 10000 # @param {type:"integer"} 
collect_steps_per_iteration = 1 # @param {type:"integer"}
replay_buffer_capacity = 1000000 # @param {type:"integer"}

batch_size = 256 # @param {type:"integer"}

critic_learning_rate = 3e-4 # @param {type:"number"}
actor_learning_rate = 3e-4 # @param {type:"number"}
alpha_learning_rate = 3e-4 # @param {type:"number"}
target_update_tau = 0.005 # @param {type:"number"}
target_update_period = 1 # @param {type:"number"}
gamma = 0.99 # @param {type:"number"}
reward_scale_factor = 1.0 # @param {type:"number"}
gradient_clipping = None # @param

actor_fc_layer_params = (256, 256)
critic_joint_fc_layer_params = (256, 256)

log_interval = 5000 # @param {type:"integer"}

num_eval_episodes = 30 # @param {type:"integer"}
eval_interval = 10000 # @param {type:"integer"}
 

环境

RL中的环境代表了我们要解决的任务或问题。使用suites可以在TF-Agents中轻松创建标准环境。给定字符串环境名称,我们提供了不同的suites来加载来自OpenAI Gym,Atari,DM Control等来源的环境。

现在,让我们从Pybullet套件中加载Minituar环境。

 env = suite_pybullet.load(env_name)
env.reset()
PIL.Image.fromarray(env.render())
 
current_dir=/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/pybullet_envs/bullet
urdf_root=/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/pybullet_data
options= 

/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/gym/logger.py:30: UserWarning: WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))

png

在这种环境下,代理商的目标是训练一个策略,该策略将控制Minitaur机器人并使它尽可能快地前进。情节持续了1000步,而回报将是整个情节的奖励总和。

让我们看一下环境提供的信息,以observation该策略将用于生成actions

 print('Observation Spec:')
print(env.time_step_spec().observation)
print('Action Spec:')
print(env.action_spec())
 
Observation Spec:
BoundedArraySpec(shape=(28,), dtype=dtype('float32'), name='observation', minimum=[  -3.1515927   -3.1515927   -3.1515927   -3.1515927   -3.1515927
   -3.1515927   -3.1515927   -3.1515927 -167.72488   -167.72488
 -167.72488   -167.72488   -167.72488   -167.72488   -167.72488
 -167.72488     -5.71        -5.71        -5.71        -5.71
   -5.71        -5.71        -5.71        -5.71        -1.01
   -1.01        -1.01        -1.01     ], maximum=[  3.1515927   3.1515927   3.1515927   3.1515927   3.1515927   3.1515927
   3.1515927   3.1515927 167.72488   167.72488   167.72488   167.72488
 167.72488   167.72488   167.72488   167.72488     5.71        5.71
   5.71        5.71        5.71        5.71        5.71        5.71
   1.01        1.01        1.01        1.01     ])
Action Spec:
BoundedArraySpec(shape=(8,), dtype=dtype('float32'), name='action', minimum=-1.0, maximum=1.0)

正如我们所看到的,观察是相当复杂的。我们收到了28个值,分别代表所有电动机的角度,速度和转矩。作为回报,环境期望[-1, 1]之间的动作有8个值。这些是所需的电机角度。

通常,我们创建两种环境:一种用于培训,一种用于评估。大多数环境都是用纯Python编写的,但是可以使用TFPyEnvironment包装器将其轻松转换为TensorFlow。原来环境的API使用numpy的阵列中, TFPyEnvironment从这些转换到/ Tensors ,为您更方便地互动与TensorFlow政策和代理商。

 train_py_env = suite_pybullet.load(env_name)
eval_py_env = suite_pybullet.load(env_name)

train_env = tf_py_environment.TFPyEnvironment(train_py_env)
eval_env = tf_py_environment.TFPyEnvironment(eval_py_env)
 
urdf_root=/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/pybullet_data
options= 
urdf_root=/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/pybullet_data
options= 

代理商

要创建SAC代理,我们首先需要创建它将训练的网络。 SAC是一个演员批评代理,因此我们将需要两个网络。

评论家将为我们提供Q(s,a)价值估算。就是说,它将收到输入和观察到的动作,并且会给我们估计该动作对于给定状态的效果。

 observation_spec = train_env.observation_spec()
action_spec = train_env.action_spec()
critic_net = critic_network.CriticNetwork(
    (observation_spec, action_spec),
    observation_fc_layer_params=None,
    action_fc_layer_params=None,
    joint_fc_layer_params=critic_joint_fc_layer_params)
 

我们将使用该批评家来训练actor网络,这将使我们能够根据观察结果生成动作。

ActorNetwork将预测正态分布的参数。每当我们需要采取行动时,便会根据当前观测值对这种分布进行采样。

 def normal_projection_net(action_spec,init_means_output_factor=0.1):
  return normal_projection_network.NormalProjectionNetwork(
      action_spec,
      mean_transform=None,
      state_dependent_std=True,
      init_means_output_factor=init_means_output_factor,
      std_transform=sac_agent.std_clip_transform,
      scale_distribution=True)


actor_net = actor_distribution_network.ActorDistributionNetwork(
    observation_spec,
    action_spec,
    fc_layer_params=actor_fc_layer_params,
    continuous_projection_net=normal_projection_net)
 

有了这些网络,我们现在可以实例化代理。

 global_step = tf.compat.v1.train.get_or_create_global_step()
tf_agent = sac_agent.SacAgent(
    train_env.time_step_spec(),
    action_spec,
    actor_network=actor_net,
    critic_network=critic_net,
    actor_optimizer=tf.compat.v1.train.AdamOptimizer(
        learning_rate=actor_learning_rate),
    critic_optimizer=tf.compat.v1.train.AdamOptimizer(
        learning_rate=critic_learning_rate),
    alpha_optimizer=tf.compat.v1.train.AdamOptimizer(
        learning_rate=alpha_learning_rate),
    target_update_tau=target_update_tau,
    target_update_period=target_update_period,
    td_errors_loss_fn=tf.compat.v1.losses.mean_squared_error,
    gamma=gamma,
    reward_scale_factor=reward_scale_factor,
    gradient_clipping=gradient_clipping,
    train_step_counter=global_step)
tf_agent.initialize()
 

政策规定

在TF-Agent中,策略代表RL中策略的标准概念:给定time_step产生操作或操作分布。主要方法是policy_step = policy.step(time_step) ,其中policy_step是一个命名的元组PolicyStep(action, state, info)policy_step.action是要应用于环境的actionstate代表有状态(RNN)策略的状态, info可能包含辅助信息,例如动作的日志概率。

代理包含两个策略:主策略(agent.policy)和用于数据收集的行为策略(agent.collect_policy)。对于评估/部署,我们通过用GreedyPolicy()包装主要策略来采取卑鄙的行动。

 eval_policy = greedy_policy.GreedyPolicy(tf_agent.policy)
collect_policy = tf_agent.collect_policy
 

指标与评估

用于评估政策的最常见指标是平均回报。回报是在环境中为某个情节运行策略时获得的奖励总和,我们通常将其平均化为几个情节。我们可以如下计算平均回报率。

 def compute_avg_return(environment, policy, num_episodes=5):

  total_return = 0.0
  for _ in range(num_episodes):

    time_step = environment.reset()
    episode_return = 0.0

    while not time_step.is_last():
      action_step = policy.action(time_step)
      time_step = environment.step(action_step.action)
      episode_return += time_step.reward
    total_return += episode_return

  avg_return = total_return / num_episodes
  return avg_return.numpy()[0]


compute_avg_return(eval_env, eval_policy, num_eval_episodes)

# Please also see the metrics module for standard implementations of different
# metrics.
 
-0.022013525

重播缓冲区

为了跟踪从环境中收集的数据,我们将使用TFUniformReplayBuffer。使用描述要存储的张量的规范构造此重播缓冲区,可以使用tf_agent.collect_data_spec从代理获取该tf_agent.collect_data_spec

 replay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer(
    data_spec=tf_agent.collect_data_spec,
    batch_size=train_env.batch_size,
    max_length=replay_buffer_capacity)
 

对于大多数代理而言, collect_data_spec是一个名为tuple的Trajectory其中包含观察,动作,奖励等。

数据采集

现在,我们将创建一个驱动程序来收集经验,以作为重播缓冲区的种子。驱动程序为我们提供了一种使用特定策略在环境中收集n步骤或事件的简单方法。

 initial_collect_driver = dynamic_step_driver.DynamicStepDriver(
        train_env,
        collect_policy,
        observers=[replay_buffer.add_batch],
        num_steps=initial_collect_steps)
initial_collect_driver.run()
 
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tf_agents/drivers/dynamic_step_driver.py:203: calling while_loop_v2 (from tensorflow.python.ops.control_flow_ops) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.while_loop(c, b, vars, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.while_loop(c, b, vars))

(TimeStep(step_type=<tf.Tensor: shape=(1,), dtype=int32, numpy=array([1], dtype=int32)>, reward=<tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.00101085], dtype=float32)>, discount=<tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>, observation=<tf.Tensor: shape=(1, 28), dtype=float32, numpy=
 array([[  1.308478  ,   2.166422  ,   1.5081352 ,   2.0260656 ,
           2.1123457 ,   1.114552  ,   1.5866141 ,   1.524472  ,
           6.9441314 ,   6.6945276 ,  -7.403659  , -20.185253  ,
          -4.8489103 ,  -1.2003611 , -19.449749  , -16.223652  ,
           4.2634044 ,   0.371617  ,  -0.92654324,  -3.8810008 ,
          -5.7       ,   3.10348   ,  -2.9569836 ,   3.916052  ,
           0.0551226 ,   0.10631521,  -0.09753982,   0.9880003 ]],
       dtype=float32)>),
 ())

为了从重播缓冲区中采样数据,我们将创建一个tf.data管道,我们可以将其馈送给代理以供以后训练。我们可以指定sample_batch_size来配置从重播缓冲区中采样的项目数。我们还可以使用并行调用和预取来优化数据管道。

为了节省空间,我们仅将当前观察结果存储在重播缓冲区的每一行中。但是由于SAC代理需要当前和下一个观测值来计算损失,因此我们总是通过设置num_steps=2来对批次中每个项目的两个相邻行进行采样。

 # Dataset generates trajectories with shape [Bx2x...]
dataset = replay_buffer.as_dataset(
    num_parallel_calls=3, sample_batch_size=batch_size, num_steps=2).prefetch(3)

iterator = iter(dataset)
 
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/autograph/operators/control_flow.py:1004: ReplayBuffer.get_next (from tf_agents.replay_buffers.replay_buffer) is deprecated and will be removed in a future version.
Instructions for updating:
Use `as_dataset(..., single_deterministic_pass=False) instead.

培训代理商

训练循环涉及从环境中收集数据和优化座席网络。在此过程中,我们有时会评估代理商的政策以了解我们的情况。

 collect_driver = dynamic_step_driver.DynamicStepDriver(
    train_env,
    collect_policy,
    observers=[replay_buffer.add_batch],
    num_steps=collect_steps_per_iteration)
 
 
try:
  %%time
except:
  pass

# (Optional) Optimize by wrapping some of the code in a graph using TF function.
tf_agent.train = common.function(tf_agent.train)
collect_driver.run = common.function(collect_driver.run)

# Reset the train step
tf_agent.train_step_counter.assign(0)

# Evaluate the agent's policy once before training.
avg_return = compute_avg_return(eval_env, eval_policy, num_eval_episodes)
returns = [avg_return]

for _ in range(num_iterations):

  # Collect a few steps using collect_policy and save to the replay buffer.
  collect_driver.run()

  # Sample a batch of data from the buffer and update the agent's network.
  experience, unused_info = next(iterator)
  train_loss = tf_agent.train(experience)

  step = tf_agent.train_step_counter.numpy()

  if step % log_interval == 0:
    print('step = {0}: loss = {1}'.format(step, train_loss.loss))

  if step % eval_interval == 0:
    avg_return = compute_avg_return(eval_env, eval_policy, num_eval_episodes)
    print('step = {0}: Average Return = {1}'.format(step, avg_return))
    returns.append(avg_return)
 
WARNING:absl:Need to use a loss function that computes losses per sample, ex: replace losses.mean_squared_error with tf.math.squared_difference. Invalid value passed for `per_example_loss`. Expected a tensor tensor with at least rank 1, received: Tensor("critic_loss/add_1:0", shape=(), dtype=float32)
WARNING:absl:Need to use a loss function that computes losses per sample, ex: replace losses.mean_squared_error with tf.math.squared_difference. Invalid value passed for `per_example_loss`. Expected a tensor tensor with at least rank 1, received: Tensor("critic_loss/add_1:0", shape=(), dtype=float32)

step = 5000: loss = -63.16588592529297
step = 10000: loss = -61.471351623535156
step = 10000: Average Return = 0.07441557198762894
step = 15000: loss = -31.185678482055664
step = 20000: loss = -18.064279556274414
step = 20000: Average Return = -0.12959735095500946
step = 25000: loss = -15.05502986907959
step = 30000: loss = -12.023421287536621
step = 30000: Average Return = -1.4209648370742798
step = 35000: loss = -5.994253635406494
step = 40000: loss = -3.944823741912842
step = 40000: Average Return = -0.6664859652519226
step = 45000: loss = 0.3637888431549072
step = 50000: loss = -3.2982077598571777
step = 50000: Average Return = 0.0521695651113987
step = 55000: loss = -2.7744715213775635
step = 60000: loss = 1.7074693441390991
step = 60000: Average Return = -0.3222312033176422
step = 65000: loss = -1.8334136009216309
step = 70000: loss = -1.4784929752349854
step = 70000: Average Return = 0.6373701095581055
step = 75000: loss = 0.48983949422836304
step = 80000: loss = 1.5974589586257935
step = 80000: Average Return = 0.1859637051820755
step = 85000: loss = -5.309885501861572
step = 90000: loss = 0.42465153336524963
step = 90000: Average Return = 0.8508636951446533
step = 95000: loss = -6.7512335777282715
step = 100000: loss = 1.8088481426239014
step = 100000: Average Return = 0.24124357104301453

可视化

情节

我们可以绘制平均收益与整体步骤的关系图,以查看我们代理商的表现。在Minitaur ,奖励功能基于Minitaur行走1000步并惩罚能源消耗的距离。

 

steps = range(0, num_iterations + 1, eval_interval)
plt.plot(steps, returns)
plt.ylabel('Average Return')
plt.xlabel('Step')
plt.ylim()
 
(-1.5345562636852264, 0.9644551217556)

png

影片

通过在每个步骤渲染环境来可视化代理的性能很有帮助。在执行此操作之前,让我们首先创建一个将视频嵌入此colab的功能。

 def embed_mp4(filename):
  """Embeds an mp4 file in the notebook."""
  video = open(filename,'rb').read()
  b64 = base64.b64encode(video)
  tag = '''
  <video width="640" height="480" controls>
    <source src="data:video/mp4;base64,{0}" type="video/mp4">
  Your browser does not support the video tag.
  </video>'''.format(b64.decode())

  return IPython.display.HTML(tag)
 

以下代码显示了几集的代理策略:

 num_episodes = 3
video_filename = 'sac_minitaur.mp4'
with imageio.get_writer(video_filename, fps=60) as video:
  for _ in range(num_episodes):
    time_step = eval_env.reset()
    video.append_data(eval_py_env.render())
    while not time_step.is_last():
      action_step = tf_agent.policy.action(time_step)
      time_step = eval_env.step(action_step.action)
      video.append_data(eval_py_env.render())

embed_mp4(video_filename)