# tf.contrib.seq2seq.BahdanauAttention

<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py#L912-L1028) |

Implements Bahdanau-style (additive) attention.

    tf.contrib.seq2seq.BahdanauAttention(
        num_units, memory, memory_sequence_length=None, normalize=False,
        probability_fn=None, score_mask_value=None, dtype=None,
        custom_key_value_fn=None, name='BahdanauAttention'
    )

This attention has two forms. The first is Bahdanau attention,
as described in:

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio.
"Neural Machine Translation by Jointly Learning to Align and Translate."
ICLR 2015. <https://arxiv.org/abs/1409.0473>

The second is the normalized form. This form is inspired by the
weight normalization article:

Tim Salimans, Diederik P. Kingma.
"Weight Normalization: A Simple Reparameterization to Accelerate
Training of Deep Neural Networks."
<https://arxiv.org/abs/1602.07868>

To enable the second form, construct the object with parameter
`normalize=True`.

<br />

<br />

<br />

| ## Args ||
|---|---|
| `num_units` | The depth of the query mechanism. |
| `memory` | The memory to query; usually the output of an RNN encoder. This tensor should be shaped `[batch_size, max_time, ...]`. |
| `memory_sequence_length` | (optional) Sequence lengths for the batch entries in memory. If provided, the memory tensor rows are masked with zeros for values past the respective sequence lengths. |
| `normalize` | Python boolean. Whether to normalize the energy term. |
| `probability_fn` | (optional) A `callable`. Converts the score to probabilities. The default is [`tf.nn.softmax`](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/nn/softmax). Other options include [`tf.contrib.seq2seq.hardmax`](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/contrib/seq2seq/hardmax) and [`tf.contrib.sparsemax.sparsemax`](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/contrib/sparsemax/sparsemax). Its signature should be: `probabilities = probability_fn(score)`. |
| `score_mask_value` | (optional): The mask value for score before passing into `probability_fn`. The default is -inf. Only used if `memory_sequence_length` is not None. |
| `dtype` | The data type for the query and memory layers of the attention mechanism. |
| `custom_key_value_fn` | (optional): The custom function for computing keys and values. |
| `name` | Name to use when creating ops. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `alignments_size` | <br /> <br /> |
| `batch_size` | <br /> <br /> |
| `keys` | <br /> <br /> |
| `memory_layer` | <br /> <br /> |
| `query_layer` | <br /> <br /> |
| `state_size` | <br /> <br /> |
| `values` | <br /> <br /> |

<br />

## Methods

### `initial_alignments`

[View source](https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py#L191-L208)

    initial_alignments(
        batch_size, dtype
    )

Creates the initial alignment values for the `AttentionWrapper` class.

This is important for AttentionMechanisms that use the previous alignment
to calculate the alignment at the next time step (e.g. monotonic attention).

The default behavior is to return a tensor of all zeros.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch_size` | `int32` scalar, the batch_size. |
| `dtype` | The `dtype`. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A `dtype` tensor shaped `[batch_size, alignments_size]` (`alignments_size` is the values' `max_time`). ||

<br />

### `initial_state`

[View source](https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py#L210-L225)

    initial_state(
        batch_size, dtype
    )

Creates the initial state values for the `AttentionWrapper` class.

This is important for AttentionMechanisms that use the previous alignment
to calculate the alignment at the next time step (e.g. monotonic attention).

The default behavior is to return the same output as initial_alignments.

<br />

<br />

<br />

| Args ||
|---|---|
| `batch_size` | `int32` scalar, the batch_size. |
| `dtype` | The `dtype`. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A structure of all-zero tensors with shapes as described by `state_size`. ||

<br />

### `__call__`

[View source](https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py#L987-L1028)

    __call__(
        query, state
    )

Score the query based on the keys and values.

<br />

<br />

<br />

| Args ||
|---|---|
| `query` | Tensor of dtype matching `self.values` and shape `[batch_size, query_depth]`. |
| `state` | Tensor of dtype matching `self.values` and shape `[batch_size, alignments_size]` (`alignments_size` is memory's `max_time`). |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| `alignments` | Tensor of dtype matching `self.values` and shape `[batch_size, alignments_size]` (`alignments_size` is memory's `max_time`). |

<br />