View source on GitHub 
Monotonic attention mechanism with Bahadanaustyle energy function.
Inherits From: AttentionMechanism
tfa.seq2seq.BahdanauMonotonicAttention(
units: tfa.types.TensorLike
,
memory: Optional[TensorLike] = None,
memory_sequence_length: Optional[TensorLike] = None,
normalize: bool = False,
sigmoid_noise: tfa.types.FloatTensorLike
= 0.0,
sigmoid_noise_seed: Optional[FloatTensorLike] = None,
score_bias_init: tfa.types.FloatTensorLike
= 0.0,
mode: str = 'parallel',
kernel_initializer: tfa.types.Constraint
= 'glorot_uniform',
dtype: tfa.types.AcceptableDTypes
= None,
name: str = 'BahdanauMonotonicAttention',
**kwargs
)
This type of attention enforces a monotonic constraint on the attention
distributions; that is once the model attends to a given point in the
memory it can't attend to any prior points at subsequence output timesteps.
It achieves this by using the _monotonic_probability_fn
instead of softmax
to construct its attention distributions. Since the attention scores are
passed through a sigmoid, a learnable scalar bias parameter is applied
after the score function and before the sigmoid. Otherwise, it is
equivalent to tfa.seq2seq.BahdanauAttention
. This approach is proposed in
Colin Raffel, MinhThang Luong, Peter J. Liu, Ron J. Weiss, Douglas Eck, "Online and LinearTime Attention by Enforcing Monotonic Alignments." ICML 2017. https://arxiv.org/abs/1704.00784
Args  

units

The depth of the query mechanism. 
memory

The memory to query; usually the output of an RNN encoder.
This tensor should be shaped [batch_size, max_time, ...] .

memory_sequence_length

(optional): Sequence lengths for the batch entries in memory. If provided, the memory tensor rows are masked with zeros for values past the respective sequence lengths. 
normalize

Python boolean. Whether to normalize the energy term. 
sigmoid_noise

Standard deviation of presigmoid noise. See the
docstring for _monotonic_probability_fn for more information.

sigmoid_noise_seed

(optional) Random seed for presigmoid noise. 
score_bias_init

Initial value for score bias scalar. It's recommended to initialize this to a negative value when the length of the memory is large. 
mode

How to compute the attention distribution. Must be one of
'recursive', 'parallel', or 'hard'. See the docstring for
tfa.seq2seq.monotonic_attention for more information.

kernel_initializer

(optional), the name of the initializer for the attention kernel. 
dtype

The data type for the query and memory layers of the attention mechanism. 
name

Name to use when creating ops. 
**kwargs

Dictionary that contains other common arguments for layer creation. 
Attributes  

activity_regularizer

Optional regularizer function for the output of this layer. 
alignments_size


compute_dtype

The dtype of the layer's computations.
This is equivalent to Layers automatically cast their inputs to the compute dtype, which causes
computations and the output to be in the compute dtype as well. This is done
by the base Layer class in Layers often perform certain internal computations in higher precision when

dtype

The dtype of the layer weights.
This is equivalent to 
dtype_policy

The dtype policy associated with this layer.
This is an instance of a 
dynamic

Whether the layer is dynamic (eageronly); set in the constructor. 
input

Retrieves the input tensor(s) of a layer.
Only applicable if the layer has exactly one input, i.e. if it is connected to one incoming layer. 
input_spec

InputSpec instance(s) describing the input format for this layer.
When you create a layer subclass, you can set
Now, if you try to call the layer on an input that isn't rank 4
(for instance, an input of shape
Input checks that can be specified via
For more information, see 
losses

List of losses added using the add_loss() API.
Variable regularization tensors are created when this property is accessed,
so it is eager safe: accessing

memory_initialized

Returns True if this attention mechanism has been initialized with
a memory.

metrics

List of metrics added using the add_metric() API.

name

Name of the layer (string), set in the constructor. 
name_scope

Returns a tf.name_scope instance for this class.

non_trainable_weights

List of all nontrainable weights tracked by this layer.
Nontrainable weights are not updated during training. They are expected
to be updated manually in 
output

Retrieves the output tensor(s) of a layer.
Only applicable if the layer has exactly one output, i.e. if it is connected to one incoming layer. 
state_size


submodules

Sequence of all submodules.
Submodules are modules which are properties of this module, or found as properties of modules which are properties of this module (and so on).

supports_masking

Whether this layer supports computing a mask using compute_mask .

trainable


trainable_weights

List of all trainable weights tracked by this layer.
Trainable weights are updated via gradient descent during training. 
variable_dtype

Alias of Layer.dtype , the dtype of the weights.

weights

Returns the list of all layer variables/weights. 
Methods
add_loss
add_loss(
losses, **kwargs
)
Add loss tensor(s), potentially dependent on layer inputs.
Some losses (for instance, activity regularization losses) may be dependent
on the inputs passed when calling a layer. Hence, when reusing the same
layer on different inputs a
and b
, some entries in layer.losses
may
be dependent on a
and some on b
. This method automatically keeps track
of dependencies.
This method can be used inside a subclassed layer or model's call
function, in which case losses
should be a Tensor or list of Tensors.
Example:
class MyLayer(tf.keras.layers.Layer):
def call(self, inputs):
self.add_loss(tf.abs(tf.reduce_mean(inputs)))
return inputs
This method can also be called directly on a Functional Model during
construction. In this case, any loss Tensors passed to this Model must
be symbolic and be able to be traced back to the model's Input
s. These
losses become part of the model's topology and are tracked in get_config
.
Example:
inputs = tf.keras.Input(shape=(10,))
x = tf.keras.layers.Dense(10)(inputs)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
# Activity regularization.
model.add_loss(tf.abs(tf.reduce_mean(x)))
If this is not the case for your loss (if, for example, your loss references
a Variable
of one of the model's layers), you can wrap your loss in a
zeroargument lambda. These losses are not tracked as part of the model's
topology since they can't be serialized.
Example:
inputs = tf.keras.Input(shape=(10,))
d = tf.keras.layers.Dense(10)
x = d(inputs)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
# Weight regularization.
model.add_loss(lambda: tf.reduce_mean(d.kernel))
Arguments  

losses

Loss tensor, or list/tuple of tensors. Rather than tensors, losses may also be zeroargument callables which create a loss tensor. 
**kwargs

Additional keyword arguments for backward compatibility. Accepted values: inputs  Deprecated, will be automatically inferred. 
add_metric
add_metric(
value, name=None, **kwargs
)
Adds metric tensor to the layer.
This method can be used inside the call()
method of a subclassed layer
or model.
class MyMetricLayer(tf.keras.layers.Layer):
def __init__(self):
super(MyMetricLayer, self).__init__(name='my_metric_layer')
self.mean = tf.keras.metrics.Mean(name='metric_1')
def call(self, inputs):
self.add_metric(self.mean(x))
self.add_metric(tf.reduce_sum(x), name='metric_2')
return inputs
This method can also be called directly on a Functional Model during
construction. In this case, any tensor passed to this Model must
be symbolic and be able to be traced back to the model's Input
s. These
metrics become part of the model's topology and are tracked when you
save the model via save()
.
inputs = tf.keras.Input(shape=(10,))
x = tf.keras.layers.Dense(10)(inputs)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
model.add_metric(math_ops.reduce_sum(x), name='metric_1')
inputs = tf.keras.Input(shape=(10,))
x = tf.keras.layers.Dense(10)(inputs)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
model.add_metric(tf.keras.metrics.Mean()(x), name='metric_1')
Args  

value

Metric tensor. 
name

String metric name. 
**kwargs

Additional keyword arguments for backward compatibility.
Accepted values:
aggregation  When the value tensor provided is not the result of
calling a keras.Metric instance, it will be aggregated by default
using a keras.Metric.Mean .

build
build(
input_shape
)
Creates the variables of the layer (optional, for subclass implementers).
This is a method that implementers of subclasses of Layer
or Model
can override if they need a statecreation step inbetween
layer instantiation and layer call.
This is typically used to create the weights of Layer
subclasses.
Arguments  

input_shape

Instance of TensorShape , or list of instances of
TensorShape if the layer expects a list of inputs
(one instance per input).

compute_mask
compute_mask(
inputs, mask=None
)
Computes an output mask tensor.
Arguments  

inputs

Tensor or list of tensors. 
mask

Tensor or list of tensors. 
Returns  

None or a tensor (or list of tensors, one per output tensor of the layer). 
compute_output_shape
compute_output_shape(
input_shape
)
Computes the output shape of the layer.
If the layer has not been built, this method will call build
on the
layer. This assumes that the layer will later be used with inputs that
match the input shape provided here.
Arguments  

input_shape

Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer. 
Returns  

An input shape tuple. 
count_params
count_params()
Count the total number of scalars composing the weights.
Returns  

An integer count. 
Raises  

ValueError

if the layer isn't yet built (in which case its weights aren't yet defined). 
deserialize_inner_layer_from_config
@classmethod
deserialize_inner_layer_from_config( config, custom_objects )
Helper method that reconstruct the query and memory from the config.
In the get_config() method, the query and memory layer configs are serialized into dict for persistence, this method perform the reverse action to reconstruct the layer from the config.
Args  

config

dict, the configs that will be used to reconstruct the object. 
custom_objects

dict mapping class names (or function names) of custom (nonKeras) objects to class/functions. 
Returns  

config

dict, the config with layer instance created, which is ready to be used as init parameters. 
from_config
@classmethod
from_config( config, custom_objects=None )
Creates a layer from its config.
This method is the reverse of get_config
,
capable of instantiating the same layer from the config
dictionary. It does not handle layer connectivity
(handled by Network), nor weights (handled by set_weights
).
Arguments  

config

A Python dictionary, typically the output of get_config. 
Returns  

A layer instance. 
get_config
get_config()
Returns the config of the layer.
A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.
The config of a layer does not include connectivity
information, nor the layer class name. These are handled
by Network
(one layer of abstraction above).
Returns  

Python dictionary. 
get_weights
get_weights()
Returns the current weights of the layer.
The weights of a layer represent the state of the layer. This function returns both trainable and nontrainable weight values associated with this layer as a list of Numpy arrays, which can in turn be used to load state into similarly parameterized layers.
For example, a Dense layer returns a list of two values peroutput weights and the bias value. These can be used to set the weights of another Dense layer:
a = tf.keras.layers.Dense(1,
kernel_initializer=tf.constant_initializer(1.))
a_out = a(tf.convert_to_tensor([[1., 2., 3.]]))
a.get_weights()
[array([[1.],
[1.],
[1.]], dtype=float32), array([0.], dtype=float32)]
b = tf.keras.layers.Dense(1,
kernel_initializer=tf.constant_initializer(2.))
b_out = b(tf.convert_to_tensor([[10., 20., 30.]]))
b.get_weights()
[array([[2.],
[2.],
[2.]], dtype=float32), array([0.], dtype=float32)]
b.set_weights(a.get_weights())
b.get_weights()
[array([[1.],
[1.],
[1.]], dtype=float32), array([0.], dtype=float32)]
Returns  

Weights values as a list of numpy arrays. 
initial_alignments
initial_alignments(
batch_size, dtype
)
Creates the initial alignment values for the monotonic attentions.
Initializes to dirac distributions, i.e. [1, 0, 0, ...memory length..., 0] for all entries in the batch.
Args  

batch_size

int32 scalar, the batch_size.

dtype

The dtype .

Returns  

A dtype tensor shaped [batch_size, alignments_size]
(alignments_size is the values' max_time ).

initial_state
initial_state(
batch_size, dtype
)
Creates the initial state values for the tfa.seq2seq.AttentionWrapper
class.
This is important for attention mechanisms that use the previous alignment to calculate the alignment at the next time step (e.g. monotonic attention).
The default behavior is to return the same output as
initial_alignments
.
Args  

batch_size

int32 scalar, the batch_size.

dtype

The dtype .

Returns  

A structure of allzero tensors with shapes as described by
state_size .

set_weights
set_weights(
weights
)
Sets the weights of the layer, from Numpy arrays.
The weights of a layer represent the state of the layer. This function sets the weight values from numpy arrays. The weight values should be passed in the order they are created by the layer. Note that the layer's weights must be instantiated before calling this function by calling the layer.
For example, a Dense layer returns a list of two values peroutput weights and the bias value. These can be used to set the weights of another Dense layer:
a = tf.keras.layers.Dense(1,
kernel_initializer=tf.constant_initializer(1.))
a_out = a(tf.convert_to_tensor([[1., 2., 3.]]))
a.get_weights()
[array([[1.],
[1.],
[1.]], dtype=float32), array([0.], dtype=float32)]
b = tf.keras.layers.Dense(1,
kernel_initializer=tf.constant_initializer(2.))
b_out = b(tf.convert_to_tensor([[10., 20., 30.]]))
b.get_weights()
[array([[2.],
[2.],
[2.]], dtype=float32), array([0.], dtype=float32)]
b.set_weights(a.get_weights())
b.get_weights()
[array([[1.],
[1.],
[1.]], dtype=float32), array([0.], dtype=float32)]
Arguments  

weights

a list of Numpy arrays. The number
of arrays and their shape must match
number of the dimensions of the weights
of the layer (i.e. it should match the
output of get_weights ).

Raises  

ValueError

If the provided weights list does not match the layer's specifications. 
setup_memory
setup_memory(
memory, memory_sequence_length=None, memory_mask=None
)
Preprocess the memory before actually query the memory.
This should only be called once at the first invocation of call()
.
Args  

memory

The memory to query; usually the output of an RNN encoder.
This tensor should be shaped [batch_size, max_time, ...] .
memory_sequence_length (optional): Sequence lengths for the batch
entries in memory. If provided, the memory tensor rows are masked
with zeros for values past the respective sequence lengths.

memory_mask

(Optional) The boolean tensor with shape [batch_size,
max_time] . For any value equal to False, the corresponding value
in memory should be ignored.

with_name_scope
@classmethod
with_name_scope( method )
Decorator to automatically enter the module name scope.
class MyModule(tf.Module):
@tf.Module.with_name_scope
def __call__(self, x):
if not hasattr(self, 'w'):
self.w = tf.Variable(tf.random.normal([x.shape[1], 3]))
return tf.matmul(x, self.w)
Using the above module would produce tf.Variable
s and tf.Tensor
s whose
names included the module name:
mod = MyModule()
mod(tf.ones([1, 2]))
<tf.Tensor: shape=(1, 3), dtype=float32, numpy=..., dtype=float32)>
mod.w
<tf.Variable 'my_module/Variable:0' shape=(2, 3) dtype=float32,
numpy=..., dtype=float32)>
Args  

method

The method to wrap. 
Returns  

The original method wrapped such that it enters the module's name scope. 
__call__
__call__(
inputs, **kwargs
)
Preprocess the inputs before calling base_layer.__call__()
.
Note that there are situation here, one for setup memory, and one with actual query and state.
 When the memory has not been configured, we just pass all the param
to
base_layer.__call__()
, which will then invokeself.call()
with proper inputs, which allows this class to setup memory.  When the memory has already been setup, the input should contain
query and state, and optionally processed memory. If the processed
memory is not included in the input, we will have to append it to
the inputs and give it to the
base_layer.__call__()
. The processed memory is the output of first invocation ofself.__call__()
. If we don't add it here, then from keras perspective, the graph is disconnected since the output from previous call is never used.
Args  

inputs

the inputs tensors. 
**kwargs

dict, other keyeword arguments for the __call__()
