Stay organized with collections Save and categorize content based on your preferences.

  • Description:

Datasets for the MT-Opt paper.

      title={MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale},
      author={Dmitry Kalashnikov and Jacob Varley and Yevgen Chebotar and Benjamin Swanson and Rico Jonschkowski and Chelsea Finn and Sergey Levine and Karol Hausman},

mt_opt/rlds (default config)

  • Config description: This dataset contains task episodes collected across afleet of real robots. It follows the RLDS formatto represent steps and episodes.

  • Dataset size: 4.38 TiB

  • Splits:

Split Examples
'train' 920,165
  • Feature structure:
    'episode_id': tf.string,
    'skill': tf.uint8,
    'steps': Dataset({
        'action': FeaturesDict({
            'close_gripper': tf.bool,
            'open_gripper': tf.bool,
            'target_pose': Tensor(shape=(7,), dtype=tf.float32),
            'terminate': tf.bool,
        'is_first': tf.bool,
        'is_last': tf.bool,
        'is_terminal': tf.bool,
        'observation': FeaturesDict({
            'gripper_closed': tf.bool,
            'height_to_bottom': tf.float32,
            'image': Image(shape=(512, 640, 3), dtype=tf.uint8),
            'state_dense': Tensor(shape=(7,), dtype=tf.float32),
    'task_code': tf.string,
  • Feature documentation:
Feature Class Shape Dtype Description
episode_id Tensor tf.string
skill Tensor tf.uint8
steps Dataset
steps/action FeaturesDict
steps/action/close_gripper Tensor tf.bool
steps/action/open_gripper Tensor tf.bool
steps/action/target_pose Tensor (7,) tf.float32
steps/action/terminate Tensor tf.bool
steps/is_first Tensor tf.bool
steps/is_last Tensor tf.bool
steps/is_terminal Tensor tf.bool
steps/observation FeaturesDict
steps/observation/gripper_closed Tensor tf.bool
steps/observation/height_to_bottom Tensor tf.float32
steps/observation/image Image (512, 640, 3) tf.uint8
steps/observation/state_dense Tensor (7,) tf.float32
task_code Tensor tf.string


  • Config description: The success detectors dataset that contains human curated definitions of tasks completion.

  • Dataset size: 548.56 GiB

  • Splits:

Split Examples
'test' 94,636
'train' 380,234
  • Feature structure:
    'image_0': Image(shape=(512, 640, 3), dtype=tf.uint8),
    'image_1': Image(shape=(480, 640, 3), dtype=tf.uint8),
    'image_2': Image(shape=(480, 640, 3), dtype=tf.uint8),
    'success': tf.bool,
    'task_code': tf.string,
  • Feature documentation:
Feature Class Shape Dtype Description
image_0 Image (512, 640, 3) tf.uint8
image_1 Image (480, 640, 3) tf.uint8
image_2 Image (480, 640, 3) tf.uint8
success Tensor tf.bool
task_code Tensor tf.string