opennmt.tokenizers module

Module defining tokenizers.

Tokenizers can work on string tf.Tensor as in-graph transformation.