# tf_agents.replay_buffers.ReverbReplayBuffer

<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/reverb_replay_buffer.py#L36-L457) |

Reverb ReplayBuffer exposed as a TF-Agents replay buffer.

Inherits From: [`ReplayBuffer`](https://www.tensorflow.org/agents/api_docs/python/tf_agents/replay_buffers/replay_buffer/ReplayBuffer)

#### View aliases


**Main aliases**

[`tf_agents.replay_buffers.reverb_replay_buffer.ReverbReplayBuffer`](https://www.tensorflow.org/agents/api_docs/python/tf_agents/replay_buffers/ReverbReplayBuffer)

<br />

    tf_agents.replay_buffers.ReverbReplayBuffer(
        data_spec,
        table_name,
        sequence_length,
        server_address=None,
        local_server=None,
        dataset_buffer_size=None,
        max_cycle_length=32,
        num_workers_per_iterator=-1,
        max_samples_per_stream=-1,
        rate_limiter_timeout_ms=-1
    )

### Used in the notebooks

| Used in the tutorials |
|---|
| - [Train a Deep Q Network with TF-Agents](https://www.tensorflow.org/agents/tutorials/1_dqn_tutorial) - [REINFORCE agent](https://www.tensorflow.org/agents/tutorials/6_reinforce_tutorial) - [SAC minitaur with the Actor-Learner API](https://www.tensorflow.org/agents/tutorials/7_SAC_minitaur_tutorial) |

<br />

<br />

<br />

| ## Args ||
|---|---|
| `data_spec` | Spec for the data held in the replay buffer. |
| `table_name` | Name of the table that will be sampled. |
| `sequence_length` | (can be set to `None`, i.e unknown) The number of timesteps that each sample consists of. If not `None`, then the lengths of samples received from the server will be validated against this number. **NOTE** This replay buffer will be at its most performant if the `sequence_length` here is equal to `num_steps` passed to `as_dataset`, and is also used when writing to the replay buffer (for example, see the `sequence_lengths` argument of the `Reverb.*Observer` classes). |
| `server_address` | (Optional) Address of the reverb replay server. One of `server_address` or `local_server` must be provided. |
| `local_server` | (Optional) An instance of `reverb.Server` that holds the replay's data. |
| `dataset_buffer_size` | (Optional) This is the prefetch buffer size (in number of items) of the Reverb Dataset object. A good rule of thumb is to set this value to 2-3x times the sample_batch_size you will use. |
| `max_cycle_length` | (Optional) The number of sequences used to populate the batches of `as_dataset`. By default, `min(32, sample_batch_size)` is used, but the number can be between `1` and `sample_batch_size`. |
| `num_workers_per_iterator` | (Defaults to -1, i.e auto selected) The number of worker threads to create per dataset iterator. When the selected table uses a FIFO or Heap sampler (i.e a queue) then exactly 1 worker must be used to avoid races causing invalid ordering of items. For all other samplers, this value should be roughly equal to the number of threads available on the CPU. |
| `max_samples_per_stream` | (Defaults to -1, i.e auto selected) The maximum number of samples to fetch from a stream before a new call is made. Keeping this number low ensures that the data is fetched uniformly from all servers. |
| `rate_limiter_timeout_ms` | (Defaults to -1: infinite). Timeout (in milliseconds) to wait on the rate limiter when sampling from the table. If `rate_limiter_timeout_ms >= 0`, this is the timeout passed to `Table::Sample` describing how long to wait for the rate limiter to allow sampling. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `capacity` | Returns the capacity of the replay buffer. |
| `data_spec` | Returns the spec for items in the replay buffer. |
| `local_server` | <br /> <br /> |
| `py_client` | <br /> <br /> |
| `stateful_dataset` | Returns whether the dataset of the replay buffer has stateful ops. |
| `tf_client` | <br /> <br /> |

<br />

## Methods

### `add_batch`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/reverb_replay_buffer.py#L169-L188)

    add_batch(
        items
    )

Adds a batch of items to the replay buffer.

> [!WARNING]
> **Warning:** `ReverbReplayBuffer` does not support `add_batch`. See `reverb_utils.ReverbObserver` for more information on how to add data to the buffer.

<br />

<br />

<br />

| Args ||
|---|---|
| `items` | Ignored. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| Nothing. ||

<br />

Raises: NotImplementedError

### `as_dataset`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/replay_buffer.py#L134-L243)

    as_dataset(
        sample_batch_size=None,
        num_steps=None,
        num_parallel_calls=None,
        sequence_preprocess_fn=None,
        single_deterministic_pass=False
    )

Creates and returns a dataset that returns entries from the buffer.

A single entry from the dataset is the result of the following pipeline:

- Sample sequences from the underlying data store
- (optionally) Process them with `sequence_preprocess_fn`,
- (optionally) Split them into subsequences of length `num_steps`
- (optionally) Batch them into batches of size `sample_batch_size`.

In practice, this pipeline is executed in parallel as much as possible
if `num_parallel_calls != 1`.

#### Some additional notes:

If `num_steps is None`, different replay buffers will behave differently.
For example, `TFUniformReplayBuffer` will return single time steps without
a time dimension. In contrast, e.g., `EpisodicReplayBuffer` will return
full sequences (since each sequence may be an episode of unknown length,
the outermost shape dimension will be `None`).

If `sample_batch_size is None`, no batching is performed; and there is no
outer batch dimension in the returned Dataset entries. This setting
is useful with variable episode lengths using e.g. `EpisodicReplayBuffer`,
because it allows the user to get full episodes back, and use [`tf.data`](https://www.tensorflow.org/api_docs/python/tf/data)
to build padded or truncated batches themselves.

If `single_deterministic_pass == True`, the replay buffer will make
every attempt to ensure every time step is visited once and exactly once
in a deterministic manner (though true determinism depends on the
underlying data store). Additional work may be done to ensure minibatches
do not have multiple rows from the same episode. In some cases, this
may mean arguments like `num_parallel_calls` are ignored.

<br />

<br />

<br />

| Args ||
|---|---|
| `sample_batch_size` | (Optional.) An optional batch_size to specify the number of items to return. If None (default), a single item is returned which matches the data_spec of this class (without a batch dimension). Otherwise, a batch of sample_batch_size items is returned, where each tensor in items will have its first dimension equal to sample_batch_size and the rest of the dimensions match the corresponding data_spec. |
| `num_steps` | (Optional.) Optional way to specify that sub-episodes are desired. If None (default), a batch of single items is returned. Otherwise, a batch of sub-episodes is returned, where a sub-episode is a sequence of consecutive items in the replay_buffer. The returned tensors will have first dimension equal to sample_batch_size (if sample_batch_size is not None), subsequent dimension equal to num_steps, and remaining dimensions which match the data_spec of this class. |
| `num_parallel_calls` | (Optional.) A [`tf.int32`](https://www.tensorflow.org/api_docs/python/tf#int32) scalar [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor), representing the number elements to process in parallel. If not specified, elements will be processed sequentially. |
| `sequence_preprocess_fn` | (Optional) fn for preprocessing the collected data before it is split into subsequences of length `num_steps`. Defined in [`TFAgent.preprocess_sequence`](https://www.tensorflow.org/agents/api_docs/python/tf_agents/agents/TFAgent#preprocess_sequence). Defaults to pass through. |
| `single_deterministic_pass` | Python boolean. If `True`, the dataset will return a single deterministic pass through its underlying data. **NOTE**: If the buffer is modified while a Dataset iterator is iterating over this data, the iterator may miss any new data or otherwise have subtly invalid data. |

<br />

<br />

<br />

| Returns ||
|---|---|
| A dataset of type tf.data.Dataset, elements of which are 2-tuples of: <br /> - An item or sequence of items or batch thereof - Auxiliary info for the items (i.e. ids, probs). ||

<br />

<br />

<br />

<br />

| Raises ||
|---|---|
| `NotImplementedError` | If a non-default argument value is not supported. |
| `ValueError` | If the data spec contains lists that must be converted to tuples. |

<br />

### `clear`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/replay_buffer.py#L261-L267)

    clear()

Resets the contents of replay buffer.

<br />

<br />

<br />

| Returns ||
|---|---|
| Clears the replay buffer contents. ||

<br />

### `gather_all`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/reverb_replay_buffer.py#L415-L430)

    gather_all()

Returns all the items in buffer.

> [!WARNING]
> **Warning:** `ReverbReplayBuffer` does not support `gather_all`. See `reverb_utils.ReverbObserver` for more information on how to retrieve data from the buffer.

<br />

<br />

<br />

| Returns ||
|---|---|
| Nothing. ||

<br />

<br />

<br />

<br />

| Raises ||
|---|---|
| NotImplementedError ||

<br />

### `get_next`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/reverb_replay_buffer.py#L190-L208)

    get_next(
        sample_batch_size=None, num_steps=None, time_stacked=True
    )

Returns an item or batch of items from the buffer.

> [!WARNING]
> **Warning:** `ReverbReplayBuffer` does not support `get_next`. See `reverb_utils.ReverbObserver` for more information on how to retrieve data from the buffer.

<br />

<br />

<br />

| Args ||
|---|---|
| `sample_batch_size` | Ignored. |
| `num_steps` | Ignored. |
| `time_stacked` | Ignored. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| Nothing. ||

<br />

<br />

<br />

<br />

| Raises ||
|---|---|
| NotImplementedError ||

<br />

### `get_table_info`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/reverb_replay_buffer.py#L153-L154)

    get_table_info()

### `num_frames`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/replay_buffer.py#L68-L70)

    num_frames()

Returns the number of frames in the replay buffer.

### `update_priorities`

[View source](https://github.com/tensorflow/agents/blob/v0.19.0/tf_agents/replay_buffers/reverb_replay_buffer.py#L436-L439)

    update_priorities(
        keys, priorities
    )

Updates the priorities for the given keys.