DEBLEU example with TeacherMaskSoftmaxEmbeddingHelper and Triggers

docs/code/losses.rst

@@ -68,6 +68,14 @@ Entropy
 .. autofunction:: texar.losses.sequence_entropy_with_logits
 
 
+DEBLEU
+==================
+
+:hidden:`debleu`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autofunction:: texar.losses.debleu
+
+
 Loss Utils
 ===========
 

docs/code/modules.rst

@@ -134,6 +134,11 @@ Decoders
 .. autoclass:: texar.modules.GumbelSoftmaxEmbeddingHelper
  :members:
 
+:hidden:`TeacherMaskSoftmaxEmbeddingHelper`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.modules.TeacherMaskSoftmaxEmbeddingHelper
+ :members:
+
 :hidden:`get_helper`
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. autofunction:: texar.modules.get_helper

docs/code/utils.rst

@@ -278,3 +278,21 @@ AverageRecorder
 ==========================
 .. autoclass:: texar.utils.AverageRecorder
  :members:
+
+Trigger
+==========================
+
+:hidden:`Trigger`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.utils.Trigger
+ :members:
+
+:hidden:`ScheduledStepsTrigger`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.utils.ScheduledStepsTrigger
+ :members:
+
+:hidden:`BestEverConvergenceTrigger`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: texar.utils.BestEverConvergenceTrigger
+ :members:

examples/differentiable_expected_bleu/README.md

@@ -0,0 +1,48 @@
+# Seq2seq Model #
+
+This example builds an attentional seq2seq model for machine translation trained with Differentiable Expected BLEU (DEBLEU) and Teacher Mask. See https://openreview.net/pdf?id=S1x2aiRqFX for the implemented paper.
+
+### Dataset ###
+
+ * iwslt14: The benchmark [IWSLT2014](https://sites.google.com/site/iwsltevaluation2014/home) (de-en) machine translation dataset. 
+
+Download the data with the following cmds:
+
+```bash
+python prepare_data.py --data de-en
+```
+
+### Train the model ###
+
+Train the model with the following cmd:
+
+```bash
+python differentiable_expected_bleu.py --config_model config_model_medium --config_data config_data_iwslt14_de-en --config_train config_train --expr_name iwslt14_de-en --restore_from "" --reinitialize
+```
+
+Here:
+ * `--config_model` specifies the model config. Note not to include the `.py` suffix.
+ * `--config_data` specifies the data config.
+ * `--config_train` specifies the training config.
+ * `--expr_name` specifies the experiment name. Used as the directory name to save and restore all information.
+ * `--restore_from` specifies the checkpoint path to restore from. If not specified (or an empty string is specified), the latest checkpoint in `expr_name` is restored.
+ * `--reinitialize` is a flag indicates whether to reinitialize the state of the optimizers before training and after annealing. Default is enabled.
+
+[config_model_medium.py](./config_model_medium.py) specifies a single-layer seq2seq model with Luong attention and bi-directional RNN encoder.
+
+[config_model_large.py](./config_model_large.py) specifies a seq2seq model with Luong attention, 2-layer bi-directional RNN encoder, single-layer RNN decoder, and a connector between the final state of the encoder and the initial state of the decoder. The size of this model is quite large.
+
+[config_data_iwslt14_de-en.py](./config_data_iwslt14_de-en.py) specifies the IWSLT'14 German-English dataset.
+
+[config_train.py](./config_train.py) specifies the training (including annealing) configs.
+
+## Results ##
+
+On the IWSLT'14 German-English dataset, we ran both configs for 4~5 times. Here are the average BLEU scores attained:
+
+| config | inference beam size | Cross-Entropy baseline | DEBLEU | improvement |
+| :------------------------------------------------: | :-----------------: | :--------------------: | :----: | :---------: |
+| [config_model_medium.py](./config_model_medium.py) | 1 | 26.12 | 27.40 | 1.28 |
+| [config_model_medium.py](./config_model_medium.py) | 5 | 27.03 | 27.72 | 0.70 |
+| [config_model_large.py](./config_model_large.py) | 1 | 25.24 | 26.47 | 1.23 |
+| [config_model_large.py](./config_model_large.py) | 5 | 26.33 | 26.87 | 0.54 |

examples/differentiable_expected_bleu/config_data_iwslt14_de-en.py

@@ -0,0 +1,59 @@
+source_vocab_file = 'data/iwslt14_de-en/vocab.de'
+target_vocab_file = 'data/iwslt14_de-en/vocab.en'
+
+train_0 = {
+ 'batch_size': 80,
+ 'allow_smaller_final_batch': False,
+ 'source_dataset': {
+ "files": 'data/iwslt14_de-en/train.de',
+ 'vocab_file': source_vocab_file,
+ 'max_seq_length': 50
+ },
+ 'target_dataset': {
+ 'files': 'data/iwslt14_de-en/train.en',
+ 'vocab_file': target_vocab_file,
+ 'max_seq_length': 50
+ },
+}
+
+train_1 = {
+ 'batch_size': 160,
+ 'allow_smaller_final_batch': False,
+ 'source_dataset': {
+ "files": 'data/iwslt14_de-en/train.de',
+ 'vocab_file': source_vocab_file,
+ 'max_seq_length': 50
+ },
+ 'target_dataset': {
+ 'files': 'data/iwslt14_de-en/train.en',
+ 'vocab_file': target_vocab_file,
+ 'max_seq_length': 50
+ },
+}
+
+
+val = {
+ 'batch_size': 80,
+ 'shuffle': False,
+ 'source_dataset': {
+ "files": 'data/iwslt14_de-en/valid.de',
+ 'vocab_file': source_vocab_file,
+ },
+ 'target_dataset': {
+ 'files': 'data/iwslt14_de-en/valid.en',
+ 'vocab_file': target_vocab_file,
+ },
+}
+
+test = {
+ 'batch_size': 80,
+ 'shuffle': False,
+ 'source_dataset': {
+ "files": 'data/iwslt14_de-en/test.de',
+ 'vocab_file': source_vocab_file,
+ },
+ 'target_dataset': {
+ 'files': 'data/iwslt14_de-en/test.en',
+ 'vocab_file': target_vocab_file,
+ },
+}

examples/differentiable_expected_bleu/config_data_iwslt14_en-fr.py

@@ -0,0 +1,45 @@
+source_vocab_file = 'data/iwslt14_en-fr/vocab.en'
+target_vocab_file = 'data/iwslt14_en-fr/vocab.fr'
+
+batch_size = 80
+
+train = {
+ 'batch_size': batch_size,
+ 'allow_smaller_final_batch': False,
+ 'source_dataset': {
+ "files": 'data/iwslt14_en-fr/train.en',
+ 'vocab_file': source_vocab_file,
+ 'max_seq_length': 50
+ },
+ 'target_dataset': {
+ 'files': 'data/iwslt14_en-fr/train.fr',
+ 'vocab_file': target_vocab_file,
+ 'max_seq_length': 50
+ },
+}
+
+val = {
+ 'batch_size': batch_size,
+ 'shuffle': False,
+ 'source_dataset': {
+ "files": 'data/iwslt14_en-fr/valid.en',
+ 'vocab_file': source_vocab_file,
+ },
+ 'target_dataset': {
+ 'files': 'data/iwslt14_en-fr/valid.fr',
+ 'vocab_file': target_vocab_file,
+ },
+}
+
+test = {
+ 'batch_size': batch_size,
+ 'shuffle': False,
+ 'source_dataset': {
+ "files": 'data/iwslt14_en-fr/test.en',
+ 'vocab_file': source_vocab_file,
+ },
+ 'target_dataset': {
+ 'files': 'data/iwslt14_en-fr/test.fr',
+ 'vocab_file': target_vocab_file,
+ },
+}

examples/differentiable_expected_bleu/config_model_large.py

@@ -0,0 +1,39 @@
+# Attentional Seq2seq model.
+# Hyperparameters not specified here will take the default values.
+
+num_units = 1000
+embedding_dim = 500
+
+embedder = {
+ 'dim': embedding_dim
+}
+
+encoder = {
+ 'rnn_cell_fw': {
+ 'kwargs': {
+ 'num_units': num_units
+ },
+ 'num_layers': 2
+ },
+ 'output_layer_fw': {
+ 'dropout_rate': 0
+ }
+}
+
+connector = {
+ 'activation_fn': 'tanh'
+}
+
+decoder = {
+ 'rnn_cell': {
+ 'kwargs': {
+ 'num_units': num_units
+ },
+ },
+ 'attention': {
+ 'kwargs': {
+ 'num_units': num_units,
+ },
+ 'attention_layer_size': num_units
+ }
+}

examples/differentiable_expected_bleu/config_model_medium.py

@@ -0,0 +1,40 @@
+# Attentional Seq2seq model.
+# Hyperparameters not specified here will take the default values.
+
+num_units = 256
+embedding_dim = 256
+dropout = 0.2
+
+embedder = {
+ 'dim': embedding_dim
+}
+
+encoder = {
+ 'rnn_cell_fw': {
+ 'kwargs': {
+ 'num_units': num_units
+ },
+ 'dropout': {
+ 'input_keep_prob': 1. - dropout
+ }
+ }
+}
+
+connector = None
+
+decoder = {
+ 'rnn_cell': {
+ 'kwargs': {
+ 'num_units': num_units
+ },
+ 'dropout': {
+ 'input_keep_prob': 1. - dropout
+ }
+ },
+ 'attention': {
+ 'kwargs': {
+ 'num_units': num_units,
+ },
+ 'attention_layer_size': num_units
+ }
+}

examples/differentiable_expected_bleu/config_train.py

@@ -0,0 +1,80 @@
+max_epochs = 1000
+steps_per_eval = 500
+tau = 1.
+infer_beam_width = 1
+infer_max_decoding_length = 50
+
+threshold_steps = 10000
+minimum_interval_steps = 10000
+phases = [
+ # (config_data, config_train, mask_pattern)
+ ("train_0", "xe_0", None),
+ ("train_0", "xe_1", None),
+ ("train_0", "debleu_0", (2, 2)),
+ ("train_1", "debleu_0", (4, 2)),
+ ("train_1", "debleu_1", (1, 0)),
+]
+
+train_xe_0 = {
+ "optimizer": {
+ "type": "AdamOptimizer",
+ "kwargs": {
+ "learning_rate": 1e-3
+ }
+ },
+ "gradient_clip": {
+ "type": "clip_by_global_norm",
+ "kwargs": {
+ "clip_norm": 5.
+ }
+ },
+ "name": "XE_0"
+}
+
+train_xe_1 = {
+ "optimizer": {
+ "type": "AdamOptimizer",
+ "kwargs": {
+ "learning_rate": 1e-5
+ }
+ },
+ "gradient_clip": {
+ "type": "clip_by_global_norm",
+ "kwargs": {
+ "clip_norm": 5.
+ }
+ },
+ "name": "XE_1"
+}
+
+train_debleu_0 = {
+ "optimizer": {
+ "type": "AdamOptimizer",
+ "kwargs": {
+ "learning_rate": 1e-5
+ }
+ },
+ "gradient_clip": {
+ "type": "clip_by_global_norm",
+ "kwargs": {
+ "clip_norm": 5.
+ }
+ },
+ "name": "DEBLEU_0"
+}
+
+train_debleu_1 = {
+ "optimizer": {
+ "type": "AdamOptimizer",
+ "kwargs": {
+ "learning_rate": 1e-6
+ }
+ },
+ "gradient_clip": {
+ "type": "clip_by_global_norm",
+ "kwargs": {
+ "clip_norm": 5.
+ }
+ },
+ "name": "DEBLEU_1"
+}