opennmt.layers.common module

Defines common layers.

opennmt.layers.common.embedding_lookup(params, ids)[source]

Wrapper around tf.nn.embedding_lookup.

This converts gradients of the embedding variable to tensors which allows to use of optimizers that don’t support sparse gradients (e.g. Adafactor).

Parameters:
  • params – The embedding tensor.
  • ids – The ids to lookup in params.
Returns:

A tf.Tensor, the embeddings that correspond to ids.

opennmt.layers.common.dropout(x, rate, training=None)[source]

Simple dropout layer.

class opennmt.layers.common.Dense(units, weight=None, transpose=False, **kwargs)[source]

Bases: tensorflow.python.keras.layers.core.Dense

Small tf.keras.layers.Dense extension to possibly reuse an existing weight matrix.

__init__(units, weight=None, transpose=False, **kwargs)[source]

Initializes the layer.

Parameters:
  • unit – Positive integer, dimensionality of the output space.
  • weight – The weight to reuse.
  • transpose – Whether weight should be transposed or not.
  • kwargs – Additional layers arguments.
add_weight(name, *args, **kwargs)[source]

Adds a new variable to the layer, or gets an existing one; returns it.

Parameters:
  • name – variable name.
  • shape – variable shape.
  • dtype – The type of the variable. Defaults to self.dtype or float32.
  • initializer – initializer instance (callable).
  • regularizer – regularizer instance (callable).
  • trainable – whether the variable should be part of the layer’s “trainable_variables” (e.g. variables, biases) or “non_trainable_variables” (e.g. BatchNorm mean, stddev). Note, if the current variable scope is marked as non-trainable then this parameter is ignored and any added variables are also marked as non-trainable. trainable defaults to True unless synchronization is set to ON_READ.
  • constraint – constraint instance (callable).
  • partitioner – Partitioner to be passed to the Checkpointable API.
  • use_resource – Whether to use ResourceVariable.
  • synchronization – Indicates when a distributed a variable will be aggregated. Accepted values are constants defined in the class tf.VariableSynchronization. By default the synchronization is set to AUTO and the current DistributionStrategy chooses when to synchronize. If synchronization is set to ON_READ, trainable must not be set to True.
  • aggregation – Indicates how a distributed variable will be aggregated. Accepted values are constants defined in the class tf.VariableAggregation.
  • **kwargs – Additional keyword arguments. Accepted values are getter and collections.
Returns:

The created variable. Usually either a Variable or ResourceVariable instance. If partitioner is not None, a PartitionedVariable instance is returned.

Raises:
  • RuntimeError – If called with partioned variable regularization and eager execution is enabled.
  • ValueError – When giving unsupported dtype and no initializer or when trainable has been set to True with synchronization set as ON_READ.
call(inputs)[source]

This is where the layer’s logic lives.

Parameters:
  • inputs – Input tensor, or list/tuple of input tensors.
  • **kwargs – Additional keyword arguments.
Returns:

A tensor or list/tuple of tensors.

class opennmt.layers.common.LayerNorm(epsilon=1e-06, **kwargs)[source]

Bases: tensorflow.python.keras.engine.base_layer.Layer

Layer normalization.

Note

Requires TensorFlow 2.0.

__init__(epsilon=1e-06, **kwargs)[source]

Initializes this layer.

Parameters:
  • epsilon – The epsilon value to use.
  • kwargs – Additional layer arguments.
build(input_shape)[source]

Creates the variables.

call(x)[source]

Normalizes x.

class opennmt.layers.common.LayerWrapper(layer, normalize_input=False, normalize_output=False, input_dropout=0, output_dropout=0, residual_connection=False, **kwargs)[source]

Bases: tensorflow.python.keras.engine.base_layer.Layer

Layer wrapper for input/output normalization, input/output dropout and residual connection.

Note

Requires TensorFlow 2.0.

__init__(layer, normalize_input=False, normalize_output=False, input_dropout=0, output_dropout=0, residual_connection=False, **kwargs)[source]

Initializes the layer.

Parameters:
  • layer – The layer to wrap.
  • normalize_input – Apply layer normalization on the input.
  • normalize_output – Apply layer normalization on the output.
  • input_dropout – The probability to drop units in the layer input.
  • output_dropout – The probability to drop units in the layer output.
  • residual_connection – Add the inputs to layer outputs (if their shape are compatible).
  • kwargs – Additional layer arguments.
call(inputs, *args, **kwargs)[source]

Runs the wrapper.

get_config()[source]

Returns the layer wrapper configuration.

classmethod from_config(config)[source]

Creates a layer wrapper from its configuration.