seq2seq_applications.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "seq2seq_applications.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84yBvO-M1XdJ",
        "colab_type": "text"
      },
      "source": [
        "# Sequence-to-Sequence Applications\n",
        "\n",
        "## Seq2seq recap:\n",
        "- Sequence of words or tokens in, sequence of predictions out\n",
        "    - MT: Sequence of English words in, sequence of Finnish words out\n",
        "    - Compare to:\n",
        "        - Sequence classification: Sequence of items in, one prediction out\n",
        "        - Sequence Tagging/Labeling: Sequence of items in, one prediction per item out\n",
        "    \n",
        "- Seq2seq: Input and output do not need to be same length\n",
        "- **Neural Attention**\n",
        "    - Instead of a fixed-length vector representing encoded input, decoder has access to any part of the encoder state\n",
        "    - Separate context vector (ci) computed for each decoder state\n",
        "    - Decoder steps can “pay attention” to different parts of the input\n",
        "\n",
        "## Today's content (part 1):\n",
        "- How to train complicated seq2seq models with very little coding effort\n",
        "- Lemmatization as a seq2seq task\n",
        "- Other seq2seq applications\n",
        "\n",
        "## MT Frameworks/Libraries\n",
        "- MT is one of the most widely studied seq2seq problems\n",
        "- Many ready-made and well maintained libraries exist for Neural MT, e.g.\n",
        "    - OpenNMT\n",
        "    - MarianNMT\n",
        "- Developed mainly for NMT, however, these do not have any MT specific hard-coded --> can be used to any seq2seq\n",
        "\n",
        "## Why use ready-made libraries?\n",
        "- Everything already implemented (different attention models, different encoder/decoder architectures)\n",
        "- Top notch results/models very difficult to replicate (true even when code is open-source)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cbXKuNv9hVHV",
        "colab_type": "text"
      },
      "source": [
        "## Date Normalization with OpenNMT\n",
        "\n",
        "### Download and read the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rpGyQtBl6PGI",
        "colab_type": "code",
        "outputId": "b4f54e41-527a-4df9-b593-f4c6478ed183",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 715
        }
      },
      "source": [
        "!pip install OpenNMT-py"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: OpenNMT-py in /usr/local/lib/python3.6/dist-packages (1.1.1)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (0.16.0)\n",
            "Requirement already satisfied: configargparse in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (1.2.1)\n",
            "Requirement already satisfied: waitress in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (1.4.3)\n",
            "Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (1.4.0)\n",
            "Requirement already satisfied: pyonmttok==1.*; platform_system == \"Linux\" in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (1.18.3)\n",
            "Requirement already satisfied: tqdm~=4.30.0 in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (4.30.0)\n",
            "Requirement already satisfied: flask in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (1.1.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (1.12.0)\n",
            "Requirement already satisfied: tensorboard>=1.14 in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (2.2.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (3.13)\n",
            "Requirement already satisfied: torchtext==0.4.0 in /usr/local/lib/python3.6/dist-packages (from OpenNMT-py) (0.4.0)\n",
            "Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.6/dist-packages (from flask->OpenNMT-py) (2.11.2)\n",
            "Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from flask->OpenNMT-py) (7.1.1)\n",
            "Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from flask->OpenNMT-py) (1.0.1)\n",
            "Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask->OpenNMT-py) (1.1.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (46.1.3)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (3.2.1)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (0.9.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (1.6.0.post3)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (3.10.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (0.4.1)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (1.28.1)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (1.7.2)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (0.34.2)\n",
            "Requirement already satisfied: numpy>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (1.18.2)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14->OpenNMT-py) (2.21.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.10.1->flask->OpenNMT-py) (1.1.1)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14->OpenNMT-py) (1.3.0)\n",
            "Requirement already satisfied: cachetools<3.2,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14->OpenNMT-py) (3.1.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14->OpenNMT-py) (0.2.8)\n",
            "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14->OpenNMT-py) (4.0)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.14->OpenNMT-py) (2.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.14->OpenNMT-py) (2020.4.5.1)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.14->OpenNMT-py) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.14->OpenNMT-py) (1.24.3)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14->OpenNMT-py) (3.1.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=1.14->OpenNMT-py) (0.4.8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_uLzc7LfhuXX",
        "colab_type": "code",
        "outputId": "5fd79ccf-b29f-40fb-fc81-4ec27902b07e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "!wget -nc https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/generated_dates.txt"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "File ‘generated_dates.txt’ already there; not retrieving.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g_ChRkJgiSHv",
        "colab_type": "code",
        "outputId": "9ba1b7a9-0861-4274-decc-e69ba37ec9fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "def load_data(fname):\n",
        "    data = []\n",
        "    with open(fname, \"rt\", encoding=\"utf-8\") as f:\n",
        "        for line in f:\n",
        "            line = line.strip()\n",
        "            input_text, output_text = line.split(\"\\t\")\n",
        "            data.append((input_text, output_text))\n",
        "    return data\n",
        "\n",
        "data = load_data(\"generated_dates.txt\")\n",
        "\n",
        "print(\"Number of examples:\", len(data))\n",
        "print(\"First examples:\", data[:5])\n"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of examples: 100000\n",
            "First examples: [('tammikuun 18. 1987', '18.01.1987'), ('joulukuun 26. 1993', '26.12.1993'), ('KESÄKUUN 16. 2009', '16.06.2009'), ('1997/8/7', '07.08.1997'), ('9. päivänä Heinäkuuta 1981', '09.07.1981')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3u5zxVJFkACr",
        "colab_type": "text"
      },
      "source": [
        "### Create training and development files\n",
        "\n",
        "- OpenNMT\n",
        "    - command line tool, which can be used as a python library as well (but then you need to handle all steps youself)\n",
        "    - requires text files as input\n",
        "- Split data into train and development sets\n",
        "- Data format:\n",
        "    - One file for training input sequences, one file for training output sequences, line numbering must match\n",
        "    - Items (words/characters/subwords) separated by whitespace\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DQVwFIj7jzDh",
        "colab_type": "code",
        "outputId": "5927f6a4-0f2d-47ee-f7da-8fcd689abee1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        }
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "def prepare_input(data):\n",
        "  examples = []\n",
        "  for input_, output_ in data:\n",
        "    input_seq = \" \".join(c for c in input_.replace(\" \", \"@\")) # whitespace is item separator, so real whitespace must be represented differently\n",
        "    output_seq = \" \".join(c for c in output_.replace(\" \", \"@\"))\n",
        "    examples.append((input_seq, output_seq))\n",
        "  return examples\n",
        "\n",
        "\n",
        "def save_data(train_data, dev_data):\n",
        "  # write data into files\n",
        "  with open(\"train.input\", \"wt\", encoding=\"utf-8\") as train_input, open(\"train.output\", \"wt\", encoding=\"utf-8\") as train_output:\n",
        "    for input_text, output_text in train_data:\n",
        "      print(input_text, file=train_input)\n",
        "      print(output_text, file=train_output)\n",
        "\n",
        "  with open(\"dev.input\", \"wt\", encoding=\"utf-8\") as dev_input, open(\"dev.output\", \"wt\", encoding=\"utf-8\") as dev_output:\n",
        "    for input_text, output_text in dev_data:\n",
        "      print(input_text, file=dev_input)\n",
        "      print(output_text, file=dev_output)\n",
        "\n",
        "\n",
        "\n",
        "# prepare correct input representation (whitespace separated items) and save\n",
        "data = prepare_input(data)\n",
        "print(\"First examples:\", data[:5])\n",
        "\n",
        "train_data, dev_data = train_test_split(data, test_size=0.2, train_size=0.8, shuffle=True)\n",
        "save_data(train_data, dev_data)\n",
        "\n"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First examples: [('t a m m i k u u n @ 1 8 . @ 1 9 8 7', '1 8 . 0 1 . 1 9 8 7'), ('j o u l u k u u n @ 2 6 . @ 1 9 9 3', '2 6 . 1 2 . 1 9 9 3'), ('K E S Ä K U U N @ 1 6 . @ 2 0 0 9', '1 6 . 0 6 . 2 0 0 9'), ('1 9 9 7 / 8 / 7', '0 7 . 0 8 . 1 9 9 7'), ('9 . @ p ä i v ä n ä @ H e i n ä k u u t a @ 1 9 8 1', '0 9 . 0 7 . 1 9 8 1')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGicgLR2FgCD",
        "colab_type": "text"
      },
      "source": [
        "## Train"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UJ9KNaFjFnxB",
        "colab_type": "code",
        "outputId": "886f4ad0-f31b-41ec-918e-03fb6902fb61",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# run OpenNMT preprocessing, this handles vectorization etc.\n",
        "!onmt_preprocess -train_src train.input -train_tgt train.output -valid_src dev.input -valid_tgt dev.output -save_data preprocessed-data -src_words_min_frequency 5 -tgt_words_min_frequency 5 -overwrite\n",
        "\n",
        "# train model\n",
        "BATCH_SIZE = 64\n",
        "LEARNING_RATE = 0.001\n",
        "OPTIMIZER = \"adam\"\n",
        "TRAIN_STEPS = 3000 #step is one minibatch, so train for 3000 minibatches\n",
        "\n",
        "print(\"How many epochs:\", (TRAIN_STEPS*BATCH_SIZE)/len(train_data))\n",
        "\n",
        "!onmt_train -data preprocessed-data -save_model trained-model -gpu_ranks 0 -learning_rate {LEARNING_RATE} -batch_size {BATCH_SIZE} -optim {OPTIMIZER} -train_steps {TRAIN_STEPS} -save_checkpoint_steps {TRAIN_STEPS}"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2020-04-20 21:07:29,132 INFO] Extracting features...\n",
            "[2020-04-20 21:07:29,132 INFO]  * number of source features: 0.\n",
            "[2020-04-20 21:07:29,132 INFO]  * number of target features: 0.\n",
            "[2020-04-20 21:07:29,132 INFO] Building `Fields` object...\n",
            "[2020-04-20 21:07:29,132 INFO] Building & saving training data...\n",
            "[2020-04-20 21:07:29,133 WARNING] Shards for corpus train already exist, will be overwritten because `-overwrite` option is set.\n",
            "[2020-04-20 21:07:29,138 WARNING] Overwrite shards for corpus None\n",
            "[2020-04-20 21:07:29,207 INFO] Building shard 0.\n",
            "[2020-04-20 21:07:31,455 INFO]  * saving 0th train data shard to preprocessed-data.train.0.pt.\n",
            "[2020-04-20 21:07:33,149 INFO]  * tgt vocab size: 15.\n",
            "[2020-04-20 21:07:33,149 INFO]  * src vocab size: 49.\n",
            "[2020-04-20 21:07:33,151 INFO] Building & saving validation data...\n",
            "[2020-04-20 21:07:33,151 WARNING] Shards for corpus valid already exist, will be overwritten because `-overwrite` option is set.\n",
            "[2020-04-20 21:07:33,155 WARNING] Overwrite shards for corpus None\n",
            "[2020-04-20 21:07:33,183 INFO] Building shard 0.\n",
            "[2020-04-20 21:07:33,487 INFO]  * saving 0th valid data shard to preprocessed-data.valid.0.pt.\n",
            "How many epochs: 2.4\n",
            "[2020-04-20 21:07:37,196 INFO]  * src vocab size = 49\n",
            "[2020-04-20 21:07:37,196 INFO]  * tgt vocab size = 15\n",
            "[2020-04-20 21:07:37,196 INFO] Building model...\n",
            "[2020-04-20 21:07:40,417 INFO] NMTModel(\n",
            "  (encoder): RNNEncoder(\n",
            "    (embeddings): Embeddings(\n",
            "      (make_embedding): Sequential(\n",
            "        (emb_luts): Elementwise(\n",
            "          (0): Embedding(49, 500, padding_idx=1)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (rnn): LSTM(500, 500, num_layers=2, dropout=0.3)\n",
            "  )\n",
            "  (decoder): InputFeedRNNDecoder(\n",
            "    (embeddings): Embeddings(\n",
            "      (make_embedding): Sequential(\n",
            "        (emb_luts): Elementwise(\n",
            "          (0): Embedding(15, 500, padding_idx=1)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (dropout): Dropout(p=0.3, inplace=False)\n",
            "    (rnn): StackedLSTM(\n",
            "      (dropout): Dropout(p=0.3, inplace=False)\n",
            "      (layers): ModuleList(\n",
            "        (0): LSTMCell(1000, 500)\n",
            "        (1): LSTMCell(500, 500)\n",
            "      )\n",
            "    )\n",
            "    (attn): GlobalAttention(\n",
            "      (linear_in): Linear(in_features=500, out_features=500, bias=False)\n",
            "      (linear_out): Linear(in_features=1000, out_features=500, bias=False)\n",
            "    )\n",
            "  )\n",
            "  (generator): Sequential(\n",
            "    (0): Linear(in_features=500, out_features=15, bias=True)\n",
            "    (1): Cast()\n",
            "    (2): LogSoftmax()\n",
            "  )\n",
            ")\n",
            "[2020-04-20 21:07:40,417 INFO] encoder: 4032500\n",
            "[2020-04-20 21:07:40,417 INFO] decoder: 5773015\n",
            "[2020-04-20 21:07:40,417 INFO] * number of parameters: 9805515\n",
            "[2020-04-20 21:07:40,418 INFO] Starting training on GPU: [0]\n",
            "[2020-04-20 21:07:40,418 INFO] Start training loop and validate every 10000 steps...\n",
            "[2020-04-20 21:07:40,419 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:07:40,889 INFO] number of examples: 80000\n",
            "[2020-04-20 21:07:43,896 INFO] Step 50/ 3000; acc:  47.67; ppl:  4.65; xent: 1.54; lr: 0.00100; 15011/10122 tok/s;      3 sec\n",
            "[2020-04-20 21:07:46,684 INFO] Step 100/ 3000; acc:  76.43; ppl:  1.90; xent: 0.64; lr: 0.00100; 20298/12629 tok/s;      6 sec\n",
            "[2020-04-20 21:07:49,442 INFO] Step 150/ 3000; acc:  95.23; ppl:  1.15; xent: 0.14; lr: 0.00100; 20263/12760 tok/s;      9 sec\n",
            "[2020-04-20 21:07:52,016 INFO] Step 200/ 3000; acc:  98.83; ppl:  1.04; xent: 0.04; lr: 0.00100; 15866/13678 tok/s;     12 sec\n",
            "[2020-04-20 21:07:54,773 INFO] Step 250/ 3000; acc:  99.07; ppl:  1.04; xent: 0.04; lr: 0.00100; 19924/12768 tok/s;     14 sec\n",
            "[2020-04-20 21:07:57,405 INFO] Step 300/ 3000; acc:  99.86; ppl:  1.01; xent: 0.01; lr: 0.00100; 17000/13376 tok/s;     17 sec\n",
            "[2020-04-20 21:08:00,090 INFO] Step 350/ 3000; acc:  99.82; ppl:  1.01; xent: 0.01; lr: 0.00100; 18381/13112 tok/s;     20 sec\n",
            "[2020-04-20 21:08:02,727 INFO] Step 400/ 3000; acc:  99.92; ppl:  1.01; xent: 0.01; lr: 0.00100; 17544/13349 tok/s;     22 sec\n",
            "[2020-04-20 21:08:05,351 INFO] Step 450/ 3000; acc:  99.57; ppl:  1.02; xent: 0.02; lr: 0.00100; 17086/13416 tok/s;     25 sec\n",
            "[2020-04-20 21:08:08,062 INFO] Step 500/ 3000; acc:  99.81; ppl:  1.01; xent: 0.01; lr: 0.00100; 19292/12986 tok/s;     28 sec\n",
            "[2020-04-20 21:08:10,784 INFO] Step 550/ 3000; acc:  99.94; ppl:  1.00; xent: 0.00; lr: 0.00100; 19265/12932 tok/s;     30 sec\n",
            "[2020-04-20 21:08:13,494 INFO] Step 600/ 3000; acc:  99.99; ppl:  1.00; xent: 0.00; lr: 0.00100; 19530/12989 tok/s;     33 sec\n",
            "[2020-04-20 21:08:16,182 INFO] Step 650/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18877/13097 tok/s;     36 sec\n",
            "[2020-04-20 21:08:18,869 INFO] Step 700/ 3000; acc:  99.99; ppl:  1.00; xent: 0.00; lr: 0.00100; 18156/13097 tok/s;     38 sec\n",
            "[2020-04-20 21:08:21,566 INFO] Step 750/ 3000; acc:  99.93; ppl:  1.00; xent: 0.00; lr: 0.00100; 18896/13056 tok/s;     41 sec\n",
            "[2020-04-20 21:08:24,279 INFO] Step 800/ 3000; acc:  99.96; ppl:  1.00; xent: 0.00; lr: 0.00100; 19178/12974 tok/s;     44 sec\n",
            "[2020-04-20 21:08:27,002 INFO] Step 850/ 3000; acc:  99.99; ppl:  1.00; xent: 0.00; lr: 0.00100; 19344/12928 tok/s;     47 sec\n",
            "[2020-04-20 21:08:29,756 INFO] Step 900/ 3000; acc:  99.98; ppl:  1.00; xent: 0.00; lr: 0.00100; 20028/12781 tok/s;     49 sec\n",
            "[2020-04-20 21:08:32,478 INFO] Step 950/ 3000; acc:  99.98; ppl:  1.00; xent: 0.00; lr: 0.00100; 19564/12933 tok/s;     52 sec\n",
            "[2020-04-20 21:08:35,135 INFO] Step 1000/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18233/13247 tok/s;     55 sec\n",
            "[2020-04-20 21:08:37,845 INFO] Step 1050/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18749/12991 tok/s;     57 sec\n",
            "[2020-04-20 21:08:40,549 INFO] Step 1100/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18585/13021 tok/s;     60 sec\n",
            "[2020-04-20 21:08:43,265 INFO] Step 1150/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19082/12957 tok/s;     63 sec\n",
            "[2020-04-20 21:08:46,019 INFO] Step 1200/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19853/12786 tok/s;     66 sec\n",
            "[2020-04-20 21:08:48,808 INFO] Step 1250/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20925/12619 tok/s;     68 sec\n",
            "[2020-04-20 21:08:48,811 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:08:49,398 INFO] number of examples: 80000\n",
            "[2020-04-20 21:08:52,406 INFO] Step 1300/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 14510/9784 tok/s;     72 sec\n",
            "[2020-04-20 21:08:55,168 INFO] Step 1350/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20487/12746 tok/s;     75 sec\n",
            "[2020-04-20 21:08:57,924 INFO] Step 1400/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20279/12770 tok/s;     78 sec\n",
            "[2020-04-20 21:09:00,509 INFO] Step 1450/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 15799/13620 tok/s;     80 sec\n",
            "[2020-04-20 21:09:03,250 INFO] Step 1500/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20040/12842 tok/s;     83 sec\n",
            "[2020-04-20 21:09:05,873 INFO] Step 1550/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 17060/13424 tok/s;     85 sec\n",
            "[2020-04-20 21:09:08,549 INFO] Step 1600/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18436/13152 tok/s;     88 sec\n",
            "[2020-04-20 21:09:11,198 INFO] Step 1650/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 17469/13292 tok/s;     91 sec\n",
            "[2020-04-20 21:09:13,825 INFO] Step 1700/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 17066/13400 tok/s;     93 sec\n",
            "[2020-04-20 21:09:16,535 INFO] Step 1750/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19297/12989 tok/s;     96 sec\n",
            "[2020-04-20 21:09:19,248 INFO] Step 1800/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19325/12972 tok/s;     99 sec\n",
            "[2020-04-20 21:09:21,983 INFO] Step 1850/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19359/12875 tok/s;    102 sec\n",
            "[2020-04-20 21:09:24,673 INFO] Step 1900/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18857/13083 tok/s;    104 sec\n",
            "[2020-04-20 21:09:27,350 INFO] Step 1950/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18231/13151 tok/s;    107 sec\n",
            "[2020-04-20 21:09:30,043 INFO] Step 2000/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18921/13073 tok/s;    110 sec\n",
            "[2020-04-20 21:09:32,756 INFO] Step 2050/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19180/12976 tok/s;    112 sec\n",
            "[2020-04-20 21:09:35,477 INFO] Step 2100/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19354/12935 tok/s;    115 sec\n",
            "[2020-04-20 21:09:38,224 INFO] Step 2150/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20080/12815 tok/s;    118 sec\n",
            "[2020-04-20 21:09:40,953 INFO] Step 2200/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19513/12899 tok/s;    121 sec\n",
            "[2020-04-20 21:09:43,623 INFO] Step 2250/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18150/13187 tok/s;    123 sec\n",
            "[2020-04-20 21:09:46,325 INFO] Step 2300/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18802/13027 tok/s;    126 sec\n",
            "[2020-04-20 21:09:49,004 INFO] Step 2350/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18756/13141 tok/s;    129 sec\n",
            "[2020-04-20 21:09:51,712 INFO] Step 2400/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19139/12996 tok/s;    131 sec\n",
            "[2020-04-20 21:09:54,472 INFO] Step 2450/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19805/12755 tok/s;    134 sec\n",
            "[2020-04-20 21:09:57,264 INFO] Step 2500/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20908/12609 tok/s;    137 sec\n",
            "[2020-04-20 21:09:57,267 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:09:57,721 INFO] number of examples: 80000\n",
            "[2020-04-20 21:10:00,692 INFO] Step 2550/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 15229/10269 tok/s;    140 sec\n",
            "[2020-04-20 21:10:03,459 INFO] Step 2600/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20449/12723 tok/s;    143 sec\n",
            "[2020-04-20 21:10:06,209 INFO] Step 2650/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20327/12800 tok/s;    146 sec\n",
            "[2020-04-20 21:10:08,785 INFO] Step 2700/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 15853/13666 tok/s;    148 sec\n",
            "[2020-04-20 21:10:11,529 INFO] Step 2750/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 20018/12828 tok/s;    151 sec\n",
            "[2020-04-20 21:10:14,154 INFO] Step 2800/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 17046/13412 tok/s;    154 sec\n",
            "[2020-04-20 21:10:16,825 INFO] Step 2850/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 18472/13177 tok/s;    156 sec\n",
            "[2020-04-20 21:10:19,457 INFO] Step 2900/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 17580/13377 tok/s;    159 sec\n",
            "[2020-04-20 21:10:22,070 INFO] Step 2950/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 17157/13472 tok/s;    162 sec\n",
            "[2020-04-20 21:10:24,774 INFO] Step 3000/ 3000; acc: 100.00; ppl:  1.00; xent: 0.00; lr: 0.00100; 19340/13018 tok/s;    164 sec\n",
            "[2020-04-20 21:10:24,775 INFO] Saving checkpoint trained-model_step_3000.pt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HKS5ED06JR5w",
        "colab_type": "text"
      },
      "source": [
        "## Predict\n",
        "\n",
        "- Prediction also requires input files, let's write files on-the-fly with bash\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a65snhizKIBw",
        "colab_type": "code",
        "outputId": "a5c06549-7e56-4c3c-a804-c8afe557c5d4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        }
      },
      "source": [
        "#'1. helmikuuta 2003', 'Toukokuun 7. päivä 1995', '9. päivä huhtikuuta 2020'\n",
        "!echo \"1. helmikuuta 2003\" | perl -pe 's/ /@/g' | perl -CS -pe 's/(.)/\\1 /g' | tee \"tmp.tmp\"; onmt_translate -model trained-model_step_3000.pt -src tmp.tmp -output pred.txt ; cat pred.txt | perl -pe 's/ //g'\n",
        "!echo \"Toukokuun 7. päivä 1995\" | perl -pe 's/ /@/g' | perl -CS -pe 's/(.)/\\1 /g' | tee \"tmp.tmp\"; onmt_translate -model trained-model_step_3000.pt -src tmp.tmp -output pred.txt ; cat pred.txt | perl -pe 's/ //g'\n",
        "!echo \"9. päivä huhtikuuta 2020\" | perl -pe 's/ /@/g' | perl -CS -pe 's/(.)/\\1 /g' | tee \"tmp.tmp\"; onmt_translate -model trained-model_step_3000.pt -src tmp.tmp -output pred.txt ; cat pred.txt | perl -pe 's/ //g'"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1 . @ h e l m i k u u t a @ 2 0 0 3 \n",
            "[2020-04-20 21:10:29,925 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0000, PRED PPL: 1.0000\n",
            "01.02.2003\n",
            "T o u k o k u u n @ 7 . @ p ä i v ä @ 1 9 9 5 \n",
            "[2020-04-20 21:10:33,350 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0000, PRED PPL: 1.0000\n",
            "07.05.1995\n",
            "9 . @ p ä i v ä @ h u h t i k u u t a @ 2 0 2 0 \n",
            "[2020-04-20 21:10:37,009 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0000, PRED PPL: 1.0000\n",
            "09.04.2020\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTeAKPN461-0",
        "colab_type": "text"
      },
      "source": [
        "# Recap: Running OpenNMT-py\n",
        "- You need to create input sequence and output sequence text files\n",
        "    - Line numbers must match\n",
        "    - Items (characters/tokens) separated using whitespace\n",
        "- Model type can be defined using command-line parameters\n",
        "\n",
        "# Lemmatization\n",
        "\n",
        "- For the given word (which may be inflected), determine its base form (dictionary form)\n",
        "    - dogs --> dog\n",
        "    - played --> play\n",
        "    - talossa -> talo\n",
        "    - lukisimme --> lukea\n",
        "    - öiden --> yö\n",
        "\n",
        "- In some languages words can heavily change when inflecting (it's not just adding suffixes)\n",
        "- Exluding irregular words, inflections are not arbitrary, they follow certain language rules\n",
        "    - Rules can be very complicated, but learnable\n",
        "- Good fit for sequence to sequence models\n",
        "- Simple approach: inflected word in --> lemma out\n",
        "    - character level model where one character is one input unit\n",
        "    - d o g s --> d o g \n",
        "    - l u k i s i m m e --> l u k e a\n",
        "\n",
        "- Ambiguity:\n",
        "    - lives --> live (VERB) or life (NOUN)\n",
        "    - koirasta --> koira (Case=Ela) or koiras (Case=Par) or koi#rasta (Case=Nom, if such exists?)\n",
        "    - We need context representation!\n",
        "    - \"En pidä hänen koirasta, koska se haukkuu liikaa.\" --> koira\n",
        "\n",
        "- Context representations\n",
        "  - Context as sliding window of text\n",
        "      - p i d ä @ h ä n e n @ < k o i r a s t a > , @ k o s k a --> k o i r a\n",
        "      - Context representation is very sparse\n",
        "      - At the same time you need to learn how inflections work in Finnish, and to understand the context in order to generate the correct lemma\n",
        "      - Works reasonably well if you have **a lot** of training data\n",
        "  - **Context as morphological tags**\n",
        "      - l i v e s VERB Mood=Ind Number=Sing Person=3 Tense=Pres VerbForm=Fin --> l i v e\n",
        "      - l i v e s NOUN Number=Plur --> l i f e \n",
        "      - k o i r a s t a NOUN Case=Ela Number=Sing --> k o i r a\n",
        "      - k o i r a s t a NOUN Case=Par Number=Sing --> k o i r a s\n",
        "      - Compact context representation (better generalization)\n",
        "      - If you already know these morphological tags (by running a tagger), you only need to learn how inflections work\n",
        "      - Works very well also with less training data\n",
        "\n",
        "- **Brain-teaser:** Why words are not suitable input and output units in lemmatization as such?\n",
        "    - Lukisimme koko päivän kirjaa . --> lukea koko päivä kirja . (sequence labeling task, label set = lemma vocabulary)\n",
        "    - You would need to learn a mapping between words and possible lemmas for it\n",
        "      - if you see 'koirasta' in input, remember that possible lemmas are 'koira' and 'koiras', and predict one of these 'labels' based on context\n",
        "    - Vocabulary is huge, and data is sparse\n",
        "    - You would not be able to predict anything for unknown words (words not seen during training)\n",
        "      - In practise, the model would pick a (random) lemma out of all lemmas, or predict  label unknown if trained to do so\n",
        "    - \"Herra Růžičkalla on uudenaikainen moottorivene .\" --> \"herra UNK olla UNK UNK .\"\n",
        "    - However, you can transform this into reasonable sequence labeling task by predicting e.g. edit-rules but this is another story...\n",
        "\n",
        "\n",
        "## Let's train a Finnish seq2seq lemmatizer with morphological tags as context\n",
        "\n",
        "- When we have the lemmatizer model trained, we can lemmatize new text by first running a tagger model to predict morphological tags, then run the lemmatizer for each word \n",
        "\n",
        "## Download data\n",
        "\n",
        "- **treebank:** an annotated collection of text including annotation for text segmentation, part-of-speech and morphological features, lemmatization and syntactic relations\n",
        "- Finnish: https://github.com/UniversalDependencies/UD_Finnish-TDT\n",
        "- Other languages: https://universaldependencies.org\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vpq2NKiX8zag",
        "colab_type": "code",
        "outputId": "8a3729b0-430b-46bc-abf8-b5a781bd259f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 421
        }
      },
      "source": [
        "!wget https://raw.githubusercontent.com/UniversalDependencies/UD_Finnish-TDT/master/fi_tdt-ud-train.conllu\n",
        "!wget https://raw.githubusercontent.com/UniversalDependencies/UD_Finnish-TDT/master/fi_tdt-ud-dev.conllu"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-04-20 21:10:39--  https://raw.githubusercontent.com/UniversalDependencies/UD_Finnish-TDT/master/fi_tdt-ud-train.conllu\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 13443822 (13M) [text/plain]\n",
            "Saving to: ‘fi_tdt-ud-train.conllu.2’\n",
            "\n",
            "\rfi_tdt-ud-train.con   0%[                    ]       0  --.-KB/s               \rfi_tdt-ud-train.con 100%[===================>]  12.82M  --.-KB/s    in 0.05s   \n",
            "\n",
            "2020-04-20 21:10:39 (253 MB/s) - ‘fi_tdt-ud-train.conllu.2’ saved [13443822/13443822]\n",
            "\n",
            "--2020-04-20 21:10:42--  https://raw.githubusercontent.com/UniversalDependencies/UD_Finnish-TDT/master/fi_tdt-ud-dev.conllu\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1511949 (1.4M) [text/plain]\n",
            "Saving to: ‘fi_tdt-ud-dev.conllu.1’\n",
            "\n",
            "fi_tdt-ud-dev.conll 100%[===================>]   1.44M  --.-KB/s    in 0.03s   \n",
            "\n",
            "2020-04-20 21:10:42 (51.4 MB/s) - ‘fi_tdt-ud-dev.conllu.1’ saved [1511949/1511949]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Y2_BN8lxSnh",
        "colab_type": "text"
      },
      "source": [
        "## Data preprocessing\n",
        "\n",
        "What we need to do is:\n",
        "- Extract words, lemmas and morphological features\n",
        "- Create an **input sequence file** having the words (one character is one input item) and morphological features (one tag is one input item), one word per line\n",
        "- Create an **output sequence file** having the lemma (one character is one output item), one lemma per line\n",
        "- Line numbering must match\n",
        "\n",
        "**Task setting:** Given an input sequence, predict the corresponding output sequence"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_l7Uj0sq7GRW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# helper functions\n",
        "\n",
        "\"\"\"\n",
        "# sent_id = b101.1\n",
        "# text = Kävelyreitti III\n",
        "1 Kävelyreitti  kävely#reitti\tNOUN\tN\tCase=Nom|Number=Sing\t0\troot\t0:root\t_\n",
        "2\tIII\tIII\tADJ\tNum\tNumType=Ord\t1\tamod\t1:amod\t_\n",
        "\n",
        "# sent_id = b101.2\n",
        "# text = Jäällä kävely avaa aina hauskoja ja erikoisia näkökulmia kaupunkiin.\n",
        "1\tJäällä\tjää\tNOUN\tN\tCase=Ade|Number=Sing\t2\tnmod\t2:nmod\t_\n",
        "2\tkävely\tkävely\tNOUN\tN\tCase=Nom|Derivation=U|Number=Sing\t3\tnsubj\t3:nsubj\t_\n",
        "3\tavaa\tavata\tVERB\tV\tMood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin|Voice=Act\t0\troot\t0:root\t_\n",
        "4\taina\taina\tADV\tAdv\t_\t3\tadvmod\t3:advmod\t_\n",
        "5\thauskoja\thauska\tADJ\tA\tCase=Par|Degree=Pos|Number=Plur\t8\tamod\t8:amod\t_\n",
        "6\tja\tja\tCCONJ\tC\t_\t7\tcc\t7:cc\t_\n",
        "7\terikoisia\terikoinen\tADJ\tA\tCase=Par|Degree=Pos|Derivation=Inen|Number=Plur\t5\tconj\t5:conj|8:amod\t_\n",
        "8\tnäkökulmia\tnäkö#kulma\tNOUN\tN\tCase=Par|Number=Plur\t3\tobj\t3:obj\t_\n",
        "9\tkaupunkiin\tkaupunki\tNOUN\tN\tCase=Ill|Number=Sing\t8\tnmod\t8:nmod\tSpaceAfter=No\n",
        "10\t.\t.\tPUNCT\tPunct\t_\t3\tpunct\t3:punct\t_\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "ID, FORM, LEMMA, UPOS, XPOS, FEATS, HEAD, DEPREL, DEPS, MISC = range(10)\n",
        "\n",
        "def read_conllu(f):\n",
        "    sent=[]\n",
        "    comment=[]\n",
        "    for line in f:\n",
        "        line=line.strip()\n",
        "        if not line: # new sentence\n",
        "            if sent:\n",
        "                yield comment,sent\n",
        "            comment=[]\n",
        "            sent=[]\n",
        "        elif line.startswith(\"#\"):\n",
        "            comment.append(line)\n",
        "        else: #normal line\n",
        "            sent.append(line.split(\"\\t\"))\n",
        "    else:\n",
        "        if sent:\n",
        "            yield comment, sent\n",
        "\n",
        "\n",
        "def prepare_lemmatization_dataset(fname):\n",
        "  # create input and output sequences (in text format)\n",
        "  data = []\n",
        "  with open(fname, \"rt\", encoding=\"utf-8\") as f:\n",
        "    for comments, sent in read_conllu(f):\n",
        "      for token in sent:\n",
        "        if \"-\" in token[ID] or \".\" in token[ID]: # multiword token or null node --> skip\n",
        "          continue\n",
        "        input_chars = \" \".join(c for c in token[FORM]) # our translation model uses whitespace tokenization, so let's create character level model by inserting whitespaces\n",
        "        features = \" \".join([token[UPOS]] + token[FEATS].split(\"|\")) # add morphological features\n",
        "        input_chars = input_chars + \" \" + features\n",
        "        lemma_chars = \" \".join(c for c in token[LEMMA])\n",
        "        data.append((input_chars, lemma_chars))\n",
        "  return data\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UTgNuPGywrsv",
        "colab_type": "code",
        "outputId": "961a87f2-85f3-4596-9119-635010a4032c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 147
        }
      },
      "source": [
        "import random \n",
        "\n",
        "# transform data\n",
        "train_data = prepare_lemmatization_dataset(\"fi_tdt-ud-train.conllu\")\n",
        "random.shuffle(train_data)\n",
        "\n",
        "dev_data = prepare_lemmatization_dataset(\"fi_tdt-ud-dev.conllu\")\n",
        "\n",
        "print(\"First train examples:\", train_data[:5])\n",
        "print(\"Number of training examples:\", len(train_data))\n",
        "print(\"Number of unique training examples:\", len(set(train_data)))\n",
        "print(\"\\nNumber of development examples:\", len(dev_data))\n",
        "print(\"Number of unique development examples:\", len(set(dev_data)))\n",
        "\n",
        "save_data(train_data, dev_data)"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First train examples: [('j o h t a j a NOUN Case=Nom Derivation=Ja Number=Sing', 'j o h t a j a'), ('k o l m e NUM Case=Nom Number=Sing NumType=Card', 'k o l m e'), ('2 0 0 5 NUM NumType=Card', '2 0 0 5'), ('o s a l t a NOUN Case=Abl Number=Sing', 'o s a'), ('o l o s u h t e i d e n NOUN Case=Gen Number=Plur', 'o l o # s u h t e e t')]\n",
            "Number of training examples: 162816\n",
            "Number of unique training examples: 51100\n",
            "\n",
            "Number of development examples: 18308\n",
            "Number of unique development examples: 8662\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bJ48fQs8NEvD",
        "colab_type": "code",
        "outputId": "3b65b6ed-c3e4-4f3b-c669-a23cd401dc8e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# run preprocessing, this handles vectorization etc.\n",
        "!onmt_preprocess -train_src train.input -train_tgt train.output -valid_src dev.input -valid_tgt dev.output -save_data preprocessed-data -src_words_min_frequency 5 -tgt_words_min_frequency 5 -overwrite\n",
        "\n",
        "# train model\n",
        "\n",
        "BATCH_SIZE = 64\n",
        "LEARNING_RATE = 0.001\n",
        "OPTIMIZER = \"adam\"\n",
        "ENCODER = \"brnn\"\n",
        "TRAIN_STEPS = 7000 #step is one minibatch, so train for 7000 minibatches\n",
        "\n",
        "print(\"How many epochs:\", (TRAIN_STEPS*BATCH_SIZE)/len(train_data))\n",
        "\n",
        "!onmt_train -data preprocessed-data -save_model trained-model -gpu_ranks 0 -learning_rate {LEARNING_RATE} -batch_size {BATCH_SIZE} -optim {OPTIMIZER} -train_steps {TRAIN_STEPS} -save_checkpoint_steps {TRAIN_STEPS} -encoder_type {ENCODER}"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2020-04-20 21:10:48,747 INFO] Extracting features...\n",
            "[2020-04-20 21:10:48,747 INFO]  * number of source features: 0.\n",
            "[2020-04-20 21:10:48,748 INFO]  * number of target features: 0.\n",
            "[2020-04-20 21:10:48,748 INFO] Building `Fields` object...\n",
            "[2020-04-20 21:10:48,748 INFO] Building & saving training data...\n",
            "[2020-04-20 21:10:48,748 WARNING] Shards for corpus train already exist, will be overwritten because `-overwrite` option is set.\n",
            "[2020-04-20 21:10:48,754 WARNING] Overwrite shards for corpus None\n",
            "[2020-04-20 21:10:48,910 INFO] Building shard 0.\n",
            "[2020-04-20 21:10:53,574 INFO]  * saving 0th train data shard to preprocessed-data.train.0.pt.\n",
            "[2020-04-20 21:10:56,584 INFO]  * tgt vocab size: 115.\n",
            "[2020-04-20 21:10:56,585 INFO]  * src vocab size: 217.\n",
            "[2020-04-20 21:10:56,588 INFO] Building & saving validation data...\n",
            "[2020-04-20 21:10:56,588 WARNING] Shards for corpus valid already exist, will be overwritten because `-overwrite` option is set.\n",
            "[2020-04-20 21:10:56,592 WARNING] Overwrite shards for corpus None\n",
            "[2020-04-20 21:10:56,620 INFO] Building shard 0.\n",
            "[2020-04-20 21:10:56,930 INFO]  * saving 0th valid data shard to preprocessed-data.valid.0.pt.\n",
            "How many epochs: 2.751572327044025\n",
            "[2020-04-20 21:11:00,064 INFO]  * src vocab size = 217\n",
            "[2020-04-20 21:11:00,064 INFO]  * tgt vocab size = 115\n",
            "[2020-04-20 21:11:00,064 INFO] Building model...\n",
            "[2020-04-20 21:11:03,227 INFO] NMTModel(\n",
            "  (encoder): RNNEncoder(\n",
            "    (embeddings): Embeddings(\n",
            "      (make_embedding): Sequential(\n",
            "        (emb_luts): Elementwise(\n",
            "          (0): Embedding(217, 500, padding_idx=1)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (rnn): LSTM(500, 250, num_layers=2, dropout=0.3, bidirectional=True)\n",
            "  )\n",
            "  (decoder): InputFeedRNNDecoder(\n",
            "    (embeddings): Embeddings(\n",
            "      (make_embedding): Sequential(\n",
            "        (emb_luts): Elementwise(\n",
            "          (0): Embedding(115, 500, padding_idx=1)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (dropout): Dropout(p=0.3, inplace=False)\n",
            "    (rnn): StackedLSTM(\n",
            "      (dropout): Dropout(p=0.3, inplace=False)\n",
            "      (layers): ModuleList(\n",
            "        (0): LSTMCell(1000, 500)\n",
            "        (1): LSTMCell(500, 500)\n",
            "      )\n",
            "    )\n",
            "    (attn): GlobalAttention(\n",
            "      (linear_in): Linear(in_features=500, out_features=500, bias=False)\n",
            "      (linear_out): Linear(in_features=1000, out_features=500, bias=False)\n",
            "    )\n",
            "  )\n",
            "  (generator): Sequential(\n",
            "    (0): Linear(in_features=500, out_features=115, bias=True)\n",
            "    (1): Cast()\n",
            "    (2): LogSoftmax()\n",
            "  )\n",
            ")\n",
            "[2020-04-20 21:11:03,227 INFO] encoder: 3116500\n",
            "[2020-04-20 21:11:03,228 INFO] decoder: 5873115\n",
            "[2020-04-20 21:11:03,228 INFO] * number of parameters: 8989615\n",
            "[2020-04-20 21:11:03,229 INFO] Starting training on GPU: [0]\n",
            "[2020-04-20 21:11:03,229 INFO] Start training loop and validate every 10000 steps...\n",
            "[2020-04-20 21:11:03,229 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:11:04,313 INFO] number of examples: 162816\n",
            "[2020-04-20 21:11:06,995 INFO] Step 50/ 7000; acc:  17.02; ppl: 32.54; xent: 3.48; lr: 0.00100; 9233/5984 tok/s;      4 sec\n",
            "[2020-04-20 21:11:08,772 INFO] Step 100/ 7000; acc:  34.81; ppl: 10.62; xent: 2.36; lr: 0.00100; 17909/10430 tok/s;      6 sec\n",
            "[2020-04-20 21:11:10,724 INFO] Step 150/ 7000; acc:  47.07; ppl:  6.59; xent: 1.89; lr: 0.00100; 17832/11012 tok/s;      7 sec\n",
            "[2020-04-20 21:11:12,625 INFO] Step 200/ 7000; acc:  66.49; ppl:  3.44; xent: 1.24; lr: 0.00100; 17574/11198 tok/s;      9 sec\n",
            "[2020-04-20 21:11:14,306 INFO] Step 250/ 7000; acc:  75.06; ppl:  2.49; xent: 0.91; lr: 0.00100; 15920/10854 tok/s;     11 sec\n",
            "[2020-04-20 21:11:16,411 INFO] Step 300/ 7000; acc:  69.45; ppl:  3.18; xent: 1.16; lr: 0.00100; 16907/11944 tok/s;     13 sec\n",
            "[2020-04-20 21:11:18,529 INFO] Step 350/ 7000; acc:  76.70; ppl:  2.45; xent: 0.89; lr: 0.00100; 16919/11125 tok/s;     15 sec\n",
            "[2020-04-20 21:11:20,274 INFO] Step 400/ 7000; acc:  84.76; ppl:  1.81; xent: 0.59; lr: 0.00100; 16872/10896 tok/s;     17 sec\n",
            "[2020-04-20 21:11:22,136 INFO] Step 450/ 7000; acc:  87.46; ppl:  1.60; xent: 0.47; lr: 0.00100; 17185/11301 tok/s;     19 sec\n",
            "[2020-04-20 21:11:23,989 INFO] Step 500/ 7000; acc:  89.26; ppl:  1.55; xent: 0.44; lr: 0.00100; 16757/11316 tok/s;     21 sec\n",
            "[2020-04-20 21:11:25,928 INFO] Step 550/ 7000; acc:  85.65; ppl:  1.76; xent: 0.57; lr: 0.00100; 17262/11413 tok/s;     23 sec\n",
            "[2020-04-20 21:11:27,843 INFO] Step 600/ 7000; acc:  89.57; ppl:  1.51; xent: 0.41; lr: 0.00100; 16530/11448 tok/s;     25 sec\n",
            "[2020-04-20 21:11:29,805 INFO] Step 650/ 7000; acc:  89.70; ppl:  1.53; xent: 0.43; lr: 0.00100; 16566/11426 tok/s;     27 sec\n",
            "[2020-04-20 21:11:31,903 INFO] Step 700/ 7000; acc:  90.04; ppl:  1.51; xent: 0.41; lr: 0.00100; 17041/11715 tok/s;     29 sec\n",
            "[2020-04-20 21:11:33,635 INFO] Step 750/ 7000; acc:  94.55; ppl:  1.22; xent: 0.20; lr: 0.00100; 16226/11140 tok/s;     30 sec\n",
            "[2020-04-20 21:11:35,574 INFO] Step 800/ 7000; acc:  94.96; ppl:  1.22; xent: 0.20; lr: 0.00100; 16953/11432 tok/s;     32 sec\n",
            "[2020-04-20 21:11:37,594 INFO] Step 850/ 7000; acc:  92.52; ppl:  1.36; xent: 0.30; lr: 0.00100; 16806/11611 tok/s;     34 sec\n",
            "[2020-04-20 21:11:39,689 INFO] Step 900/ 7000; acc:  92.86; ppl:  1.34; xent: 0.29; lr: 0.00100; 16650/11607 tok/s;     36 sec\n",
            "[2020-04-20 21:11:41,539 INFO] Step 950/ 7000; acc:  95.86; ppl:  1.17; xent: 0.16; lr: 0.00100; 16745/11453 tok/s;     38 sec\n",
            "[2020-04-20 21:11:43,285 INFO] Step 1000/ 7000; acc:  94.17; ppl:  1.31; xent: 0.27; lr: 0.00100; 16205/11188 tok/s;     40 sec\n",
            "[2020-04-20 21:11:45,168 INFO] Step 1050/ 7000; acc:  95.68; ppl:  1.20; xent: 0.18; lr: 0.00100; 17570/11335 tok/s;     42 sec\n",
            "[2020-04-20 21:11:47,037 INFO] Step 1100/ 7000; acc:  96.02; ppl:  1.17; xent: 0.15; lr: 0.00100; 16959/11410 tok/s;     44 sec\n",
            "[2020-04-20 21:11:48,957 INFO] Step 1150/ 7000; acc:  96.50; ppl:  1.16; xent: 0.15; lr: 0.00100; 17860/11463 tok/s;     46 sec\n",
            "[2020-04-20 21:11:50,972 INFO] Step 1200/ 7000; acc:  95.08; ppl:  1.22; xent: 0.20; lr: 0.00100; 17423/10998 tok/s;     48 sec\n",
            "[2020-04-20 21:11:52,928 INFO] Step 1250/ 7000; acc:  94.21; ppl:  1.27; xent: 0.24; lr: 0.00100; 17213/11492 tok/s;     50 sec\n",
            "[2020-04-20 21:11:54,998 INFO] Step 1300/ 7000; acc:  95.31; ppl:  1.21; xent: 0.19; lr: 0.00100; 17032/11519 tok/s;     52 sec\n",
            "[2020-04-20 21:11:56,844 INFO] Step 1350/ 7000; acc:  97.88; ppl:  1.09; xent: 0.08; lr: 0.00100; 17056/11371 tok/s;     54 sec\n",
            "[2020-04-20 21:11:58,660 INFO] Step 1400/ 7000; acc:  97.21; ppl:  1.11; xent: 0.10; lr: 0.00100; 17364/11082 tok/s;     55 sec\n",
            "[2020-04-20 21:12:00,463 INFO] Step 1450/ 7000; acc:  97.90; ppl:  1.08; xent: 0.08; lr: 0.00100; 17429/11109 tok/s;     57 sec\n",
            "[2020-04-20 21:12:02,179 INFO] Step 1500/ 7000; acc:  97.15; ppl:  1.13; xent: 0.12; lr: 0.00100; 16716/11109 tok/s;     59 sec\n",
            "[2020-04-20 21:12:04,006 INFO] Step 1550/ 7000; acc:  96.45; ppl:  1.17; xent: 0.16; lr: 0.00100; 18197/10919 tok/s;     61 sec\n",
            "[2020-04-20 21:12:05,747 INFO] Step 1600/ 7000; acc:  98.10; ppl:  1.08; xent: 0.08; lr: 0.00100; 16951/11076 tok/s;     63 sec\n",
            "[2020-04-20 21:12:07,488 INFO] Step 1650/ 7000; acc:  97.70; ppl:  1.10; xent: 0.10; lr: 0.00100; 17614/10878 tok/s;     64 sec\n",
            "[2020-04-20 21:12:09,483 INFO] Step 1700/ 7000; acc:  95.02; ppl:  1.23; xent: 0.21; lr: 0.00100; 17005/11618 tok/s;     66 sec\n",
            "[2020-04-20 21:12:11,182 INFO] Step 1750/ 7000; acc:  97.25; ppl:  1.13; xent: 0.12; lr: 0.00100; 15905/10486 tok/s;     68 sec\n",
            "[2020-04-20 21:12:13,086 INFO] Step 1800/ 7000; acc:  97.15; ppl:  1.14; xent: 0.13; lr: 0.00100; 17038/11448 tok/s;     70 sec\n",
            "[2020-04-20 21:12:15,101 INFO] Step 1850/ 7000; acc:  96.58; ppl:  1.15; xent: 0.14; lr: 0.00100; 16986/11446 tok/s;     72 sec\n",
            "[2020-04-20 21:12:16,907 INFO] Step 1900/ 7000; acc:  97.84; ppl:  1.09; xent: 0.08; lr: 0.00100; 16556/11234 tok/s;     74 sec\n",
            "[2020-04-20 21:12:18,748 INFO] Step 1950/ 7000; acc:  97.59; ppl:  1.10; xent: 0.10; lr: 0.00100; 17314/11246 tok/s;     76 sec\n",
            "[2020-04-20 21:12:20,638 INFO] Step 2000/ 7000; acc:  98.14; ppl:  1.08; xent: 0.07; lr: 0.00100; 17237/11376 tok/s;     77 sec\n",
            "[2020-04-20 21:12:22,442 INFO] Step 2050/ 7000; acc:  97.54; ppl:  1.12; xent: 0.11; lr: 0.00100; 16680/10827 tok/s;     79 sec\n",
            "[2020-04-20 21:12:24,323 INFO] Step 2100/ 7000; acc:  97.96; ppl:  1.09; xent: 0.09; lr: 0.00100; 16501/11615 tok/s;     81 sec\n",
            "[2020-04-20 21:12:26,171 INFO] Step 2150/ 7000; acc:  96.89; ppl:  1.16; xent: 0.15; lr: 0.00100; 16699/11346 tok/s;     83 sec\n",
            "[2020-04-20 21:12:28,120 INFO] Step 2200/ 7000; acc:  96.10; ppl:  1.18; xent: 0.17; lr: 0.00100; 16847/11714 tok/s;     85 sec\n",
            "[2020-04-20 21:12:30,319 INFO] Step 2250/ 7000; acc:  97.12; ppl:  1.13; xent: 0.12; lr: 0.00100; 17250/11972 tok/s;     87 sec\n",
            "[2020-04-20 21:12:32,315 INFO] Step 2300/ 7000; acc:  96.96; ppl:  1.14; xent: 0.13; lr: 0.00100; 17011/11558 tok/s;     89 sec\n",
            "[2020-04-20 21:12:34,291 INFO] Step 2350/ 7000; acc:  97.47; ppl:  1.11; xent: 0.11; lr: 0.00100; 17232/11651 tok/s;     91 sec\n",
            "[2020-04-20 21:12:36,062 INFO] Step 2400/ 7000; acc:  98.41; ppl:  1.06; xent: 0.06; lr: 0.00100; 16990/11206 tok/s;     93 sec\n",
            "[2020-04-20 21:12:38,127 INFO] Step 2450/ 7000; acc:  97.33; ppl:  1.12; xent: 0.12; lr: 0.00100; 16887/11916 tok/s;     95 sec\n",
            "[2020-04-20 21:12:40,037 INFO] Step 2500/ 7000; acc:  97.68; ppl:  1.10; xent: 0.09; lr: 0.00100; 17294/11490 tok/s;     97 sec\n",
            "[2020-04-20 21:12:41,594 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:12:43,053 INFO] number of examples: 162816\n",
            "[2020-04-20 21:12:43,880 INFO] Step 2550/ 7000; acc:  98.13; ppl:  1.08; xent: 0.07; lr: 0.00100; 7373/4765 tok/s;    101 sec\n",
            "[2020-04-20 21:12:45,845 INFO] Step 2600/ 7000; acc:  97.85; ppl:  1.09; xent: 0.08; lr: 0.00100; 17334/11462 tok/s;    103 sec\n",
            "[2020-04-20 21:12:47,654 INFO] Step 2650/ 7000; acc:  98.29; ppl:  1.06; xent: 0.06; lr: 0.00100; 18720/10844 tok/s;    104 sec\n",
            "[2020-04-20 21:12:49,562 INFO] Step 2700/ 7000; acc:  98.04; ppl:  1.08; xent: 0.08; lr: 0.00100; 17850/11372 tok/s;    106 sec\n",
            "[2020-04-20 21:12:51,361 INFO] Step 2750/ 7000; acc:  98.48; ppl:  1.06; xent: 0.06; lr: 0.00100; 17611/11122 tok/s;    108 sec\n",
            "[2020-04-20 21:12:53,105 INFO] Step 2800/ 7000; acc:  98.98; ppl:  1.05; xent: 0.04; lr: 0.00100; 16474/11154 tok/s;    110 sec\n",
            "[2020-04-20 21:12:55,193 INFO] Step 2850/ 7000; acc:  97.06; ppl:  1.14; xent: 0.13; lr: 0.00100; 17076/11904 tok/s;    112 sec\n",
            "[2020-04-20 21:12:57,290 INFO] Step 2900/ 7000; acc:  96.18; ppl:  1.23; xent: 0.21; lr: 0.00100; 16901/11187 tok/s;    114 sec\n",
            "[2020-04-20 21:12:59,066 INFO] Step 2950/ 7000; acc:  98.03; ppl:  1.08; xent: 0.08; lr: 0.00100; 16980/11126 tok/s;    116 sec\n",
            "[2020-04-20 21:13:00,883 INFO] Step 3000/ 7000; acc:  98.61; ppl:  1.05; xent: 0.05; lr: 0.00100; 17047/11068 tok/s;    118 sec\n",
            "[2020-04-20 21:13:02,786 INFO] Step 3050/ 7000; acc:  98.29; ppl:  1.07; xent: 0.07; lr: 0.00100; 16678/11587 tok/s;    120 sec\n",
            "[2020-04-20 21:13:04,691 INFO] Step 3100/ 7000; acc:  98.37; ppl:  1.07; xent: 0.07; lr: 0.00100; 17542/11282 tok/s;    121 sec\n",
            "[2020-04-20 21:13:06,654 INFO] Step 3150/ 7000; acc:  97.73; ppl:  1.11; xent: 0.10; lr: 0.00100; 16520/11529 tok/s;    123 sec\n",
            "[2020-04-20 21:13:08,575 INFO] Step 3200/ 7000; acc:  97.45; ppl:  1.11; xent: 0.10; lr: 0.00100; 16624/11239 tok/s;    125 sec\n",
            "[2020-04-20 21:13:10,638 INFO] Step 3250/ 7000; acc:  97.68; ppl:  1.09; xent: 0.09; lr: 0.00100; 16494/11574 tok/s;    127 sec\n",
            "[2020-04-20 21:13:12,409 INFO] Step 3300/ 7000; acc:  98.63; ppl:  1.05; xent: 0.05; lr: 0.00100; 16588/11181 tok/s;    129 sec\n",
            "[2020-04-20 21:13:14,517 INFO] Step 3350/ 7000; acc:  97.49; ppl:  1.12; xent: 0.11; lr: 0.00100; 17004/11633 tok/s;    131 sec\n",
            "[2020-04-20 21:13:16,440 INFO] Step 3400/ 7000; acc:  98.43; ppl:  1.07; xent: 0.07; lr: 0.00100; 16177/11541 tok/s;    133 sec\n",
            "[2020-04-20 21:13:18,541 INFO] Step 3450/ 7000; acc:  97.95; ppl:  1.09; xent: 0.09; lr: 0.00100; 17143/11625 tok/s;    135 sec\n",
            "[2020-04-20 21:13:20,338 INFO] Step 3500/ 7000; acc:  98.48; ppl:  1.07; xent: 0.07; lr: 0.00100; 16642/11230 tok/s;    137 sec\n",
            "[2020-04-20 21:13:22,132 INFO] Step 3550/ 7000; acc:  98.34; ppl:  1.07; xent: 0.07; lr: 0.00100; 16267/11364 tok/s;    139 sec\n",
            "[2020-04-20 21:13:23,992 INFO] Step 3600/ 7000; acc:  98.38; ppl:  1.08; xent: 0.08; lr: 0.00100; 17132/11231 tok/s;    141 sec\n",
            "[2020-04-20 21:13:25,853 INFO] Step 3650/ 7000; acc:  98.57; ppl:  1.06; xent: 0.06; lr: 0.00100; 17786/11238 tok/s;    143 sec\n",
            "[2020-04-20 21:13:27,781 INFO] Step 3700/ 7000; acc:  99.06; ppl:  1.04; xent: 0.04; lr: 0.00100; 17193/11584 tok/s;    145 sec\n",
            "[2020-04-20 21:13:29,763 INFO] Step 3750/ 7000; acc:  98.15; ppl:  1.08; xent: 0.08; lr: 0.00100; 17682/11052 tok/s;    147 sec\n",
            "[2020-04-20 21:13:31,723 INFO] Step 3800/ 7000; acc:  97.80; ppl:  1.10; xent: 0.10; lr: 0.00100; 17270/11596 tok/s;    148 sec\n",
            "[2020-04-20 21:13:33,769 INFO] Step 3850/ 7000; acc:  98.37; ppl:  1.08; xent: 0.07; lr: 0.00100; 17362/11536 tok/s;    151 sec\n",
            "[2020-04-20 21:13:35,566 INFO] Step 3900/ 7000; acc:  99.01; ppl:  1.04; xent: 0.04; lr: 0.00100; 16775/11177 tok/s;    152 sec\n",
            "[2020-04-20 21:13:37,450 INFO] Step 3950/ 7000; acc:  98.78; ppl:  1.05; xent: 0.05; lr: 0.00100; 17006/11227 tok/s;    154 sec\n",
            "[2020-04-20 21:13:39,272 INFO] Step 4000/ 7000; acc:  99.01; ppl:  1.04; xent: 0.04; lr: 0.00100; 17526/11085 tok/s;    156 sec\n",
            "[2020-04-20 21:13:41,040 INFO] Step 4050/ 7000; acc:  98.84; ppl:  1.05; xent: 0.05; lr: 0.00100; 17011/11008 tok/s;    158 sec\n",
            "[2020-04-20 21:13:42,821 INFO] Step 4100/ 7000; acc:  98.74; ppl:  1.06; xent: 0.06; lr: 0.00100; 17670/10674 tok/s;    160 sec\n",
            "[2020-04-20 21:13:44,575 INFO] Step 4150/ 7000; acc:  99.27; ppl:  1.03; xent: 0.03; lr: 0.00100; 17113/11018 tok/s;    161 sec\n",
            "[2020-04-20 21:13:46,332 INFO] Step 4200/ 7000; acc:  98.97; ppl:  1.05; xent: 0.04; lr: 0.00100; 17194/10819 tok/s;    163 sec\n",
            "[2020-04-20 21:13:48,265 INFO] Step 4250/ 7000; acc:  97.22; ppl:  1.14; xent: 0.13; lr: 0.00100; 16626/11423 tok/s;    165 sec\n",
            "[2020-04-20 21:13:50,012 INFO] Step 4300/ 7000; acc:  98.37; ppl:  1.08; xent: 0.08; lr: 0.00100; 15944/10674 tok/s;    167 sec\n",
            "[2020-04-20 21:13:51,962 INFO] Step 4350/ 7000; acc:  98.61; ppl:  1.06; xent: 0.06; lr: 0.00100; 17690/11536 tok/s;    169 sec\n",
            "[2020-04-20 21:13:53,913 INFO] Step 4400/ 7000; acc:  98.50; ppl:  1.07; xent: 0.06; lr: 0.00100; 16880/11451 tok/s;    171 sec\n",
            "[2020-04-20 21:13:55,722 INFO] Step 4450/ 7000; acc:  99.07; ppl:  1.04; xent: 0.04; lr: 0.00100; 16635/11261 tok/s;    172 sec\n",
            "[2020-04-20 21:13:57,568 INFO] Step 4500/ 7000; acc:  98.83; ppl:  1.05; xent: 0.05; lr: 0.00100; 16963/11205 tok/s;    174 sec\n",
            "[2020-04-20 21:13:59,526 INFO] Step 4550/ 7000; acc:  98.86; ppl:  1.05; xent: 0.05; lr: 0.00100; 17288/11384 tok/s;    176 sec\n",
            "[2020-04-20 21:14:01,304 INFO] Step 4600/ 7000; acc:  98.34; ppl:  1.08; xent: 0.08; lr: 0.00100; 16561/10840 tok/s;    178 sec\n",
            "[2020-04-20 21:14:03,200 INFO] Step 4650/ 7000; acc:  98.75; ppl:  1.05; xent: 0.05; lr: 0.00100; 16504/11557 tok/s;    180 sec\n",
            "[2020-04-20 21:14:05,152 INFO] Step 4700/ 7000; acc:  98.31; ppl:  1.07; xent: 0.07; lr: 0.00100; 16630/11467 tok/s;    182 sec\n",
            "[2020-04-20 21:14:07,087 INFO] Step 4750/ 7000; acc:  98.14; ppl:  1.10; xent: 0.09; lr: 0.00100; 16708/11368 tok/s;    184 sec\n",
            "[2020-04-20 21:14:09,207 INFO] Step 4800/ 7000; acc:  98.06; ppl:  1.09; xent: 0.09; lr: 0.00100; 17072/11811 tok/s;    186 sec\n",
            "[2020-04-20 21:14:11,233 INFO] Step 4850/ 7000; acc:  97.77; ppl:  1.11; xent: 0.11; lr: 0.00100; 17235/11514 tok/s;    188 sec\n",
            "[2020-04-20 21:14:13,219 INFO] Step 4900/ 7000; acc:  98.44; ppl:  1.07; xent: 0.07; lr: 0.00100; 16694/11721 tok/s;    190 sec\n",
            "[2020-04-20 21:14:15,061 INFO] Step 4950/ 7000; acc:  98.93; ppl:  1.04; xent: 0.04; lr: 0.00100; 17237/11141 tok/s;    192 sec\n",
            "[2020-04-20 21:14:17,107 INFO] Step 5000/ 7000; acc:  98.22; ppl:  1.09; xent: 0.08; lr: 0.00100; 16739/11919 tok/s;    194 sec\n",
            "[2020-04-20 21:14:18,964 INFO] Step 5050/ 7000; acc:  98.62; ppl:  1.05; xent: 0.05; lr: 0.00100; 17580/11311 tok/s;    196 sec\n",
            "[2020-04-20 21:14:20,336 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:14:21,844 INFO] number of examples: 162816\n",
            "[2020-04-20 21:14:22,910 INFO] Step 5100/ 7000; acc:  98.54; ppl:  1.06; xent: 0.05; lr: 0.00100; 7358/4848 tok/s;    200 sec\n",
            "[2020-04-20 21:14:24,870 INFO] Step 5150/ 7000; acc:  98.69; ppl:  1.05; xent: 0.05; lr: 0.00100; 17282/11425 tok/s;    202 sec\n",
            "[2020-04-20 21:14:26,668 INFO] Step 5200/ 7000; acc:  98.97; ppl:  1.04; xent: 0.04; lr: 0.00100; 18622/10731 tok/s;    203 sec\n",
            "[2020-04-20 21:14:28,620 INFO] Step 5250/ 7000; acc:  98.71; ppl:  1.06; xent: 0.06; lr: 0.00100; 17739/11297 tok/s;    205 sec\n",
            "[2020-04-20 21:14:30,331 INFO] Step 5300/ 7000; acc:  98.95; ppl:  1.05; xent: 0.05; lr: 0.00100; 17093/10884 tok/s;    207 sec\n",
            "[2020-04-20 21:14:32,178 INFO] Step 5350/ 7000; acc:  98.95; ppl:  1.05; xent: 0.05; lr: 0.00100; 16839/11259 tok/s;    209 sec\n",
            "[2020-04-20 21:14:34,292 INFO] Step 5400/ 7000; acc:  98.02; ppl:  1.10; xent: 0.09; lr: 0.00100; 17142/11968 tok/s;    211 sec\n",
            "[2020-04-20 21:14:36,265 INFO] Step 5450/ 7000; acc:  97.44; ppl:  1.15; xent: 0.14; lr: 0.00100; 16789/11048 tok/s;    213 sec\n",
            "[2020-04-20 21:14:38,090 INFO] Step 5500/ 7000; acc:  98.32; ppl:  1.08; xent: 0.08; lr: 0.00100; 16843/11286 tok/s;    215 sec\n",
            "[2020-04-20 21:14:39,962 INFO] Step 5550/ 7000; acc:  98.97; ppl:  1.04; xent: 0.04; lr: 0.00100; 17259/11286 tok/s;    217 sec\n",
            "[2020-04-20 21:14:41,829 INFO] Step 5600/ 7000; acc:  99.00; ppl:  1.05; xent: 0.04; lr: 0.00100; 16803/11439 tok/s;    219 sec\n",
            "[2020-04-20 21:14:43,720 INFO] Step 5650/ 7000; acc:  98.98; ppl:  1.05; xent: 0.05; lr: 0.00100; 17128/11398 tok/s;    220 sec\n",
            "[2020-04-20 21:14:45,661 INFO] Step 5700/ 7000; acc:  98.62; ppl:  1.07; xent: 0.06; lr: 0.00100; 16313/11289 tok/s;    222 sec\n",
            "[2020-04-20 21:14:47,609 INFO] Step 5750/ 7000; acc:  98.76; ppl:  1.06; xent: 0.05; lr: 0.00100; 16882/11418 tok/s;    224 sec\n",
            "[2020-04-20 21:14:49,681 INFO] Step 5800/ 7000; acc:  98.48; ppl:  1.06; xent: 0.06; lr: 0.00100; 17035/11705 tok/s;    226 sec\n",
            "[2020-04-20 21:14:51,421 INFO] Step 5850/ 7000; acc:  99.11; ppl:  1.04; xent: 0.04; lr: 0.00100; 15985/10935 tok/s;    228 sec\n",
            "[2020-04-20 21:14:53,505 INFO] Step 5900/ 7000; acc:  98.29; ppl:  1.08; xent: 0.08; lr: 0.00100; 17345/11776 tok/s;    230 sec\n",
            "[2020-04-20 21:14:55,474 INFO] Step 5950/ 7000; acc:  98.45; ppl:  1.07; xent: 0.07; lr: 0.00100; 16055/11370 tok/s;    232 sec\n",
            "[2020-04-20 21:14:57,544 INFO] Step 6000/ 7000; acc:  98.61; ppl:  1.06; xent: 0.06; lr: 0.00100; 17311/11931 tok/s;    234 sec\n",
            "[2020-04-20 21:14:59,327 INFO] Step 6050/ 7000; acc:  98.99; ppl:  1.04; xent: 0.04; lr: 0.00100; 16732/11233 tok/s;    236 sec\n",
            "[2020-04-20 21:15:01,102 INFO] Step 6100/ 7000; acc:  98.97; ppl:  1.04; xent: 0.04; lr: 0.00100; 16545/11231 tok/s;    238 sec\n",
            "[2020-04-20 21:15:02,997 INFO] Step 6150/ 7000; acc:  98.82; ppl:  1.05; xent: 0.05; lr: 0.00100; 16997/11403 tok/s;    240 sec\n",
            "[2020-04-20 21:15:04,831 INFO] Step 6200/ 7000; acc:  99.15; ppl:  1.03; xent: 0.03; lr: 0.00100; 17696/11227 tok/s;    242 sec\n",
            "[2020-04-20 21:15:06,785 INFO] Step 6250/ 7000; acc:  99.14; ppl:  1.04; xent: 0.04; lr: 0.00100; 17514/11237 tok/s;    244 sec\n",
            "[2020-04-20 21:15:08,801 INFO] Step 6300/ 7000; acc:  98.45; ppl:  1.08; xent: 0.07; lr: 0.00100; 17288/11527 tok/s;    246 sec\n",
            "[2020-04-20 21:15:10,781 INFO] Step 6350/ 7000; acc:  98.37; ppl:  1.08; xent: 0.08; lr: 0.00100; 17398/11483 tok/s;    248 sec\n",
            "[2020-04-20 21:15:12,772 INFO] Step 6400/ 7000; acc:  99.06; ppl:  1.05; xent: 0.04; lr: 0.00100; 16965/11431 tok/s;    250 sec\n",
            "[2020-04-20 21:15:14,595 INFO] Step 6450/ 7000; acc:  99.06; ppl:  1.04; xent: 0.04; lr: 0.00100; 17273/11186 tok/s;    251 sec\n",
            "[2020-04-20 21:15:16,404 INFO] Step 6500/ 7000; acc:  99.16; ppl:  1.03; xent: 0.03; lr: 0.00100; 16897/11062 tok/s;    253 sec\n",
            "[2020-04-20 21:15:18,243 INFO] Step 6550/ 7000; acc:  99.23; ppl:  1.03; xent: 0.03; lr: 0.00100; 17296/11157 tok/s;    255 sec\n",
            "[2020-04-20 21:15:20,027 INFO] Step 6600/ 7000; acc:  99.03; ppl:  1.04; xent: 0.04; lr: 0.00100; 16952/10907 tok/s;    257 sec\n",
            "[2020-04-20 21:15:21,791 INFO] Step 6650/ 7000; acc:  99.23; ppl:  1.03; xent: 0.03; lr: 0.00100; 17780/10706 tok/s;    259 sec\n",
            "[2020-04-20 21:15:23,586 INFO] Step 6700/ 7000; acc:  99.42; ppl:  1.03; xent: 0.03; lr: 0.00100; 17335/11013 tok/s;    260 sec\n",
            "[2020-04-20 21:15:25,373 INFO] Step 6750/ 7000; acc:  98.73; ppl:  1.06; xent: 0.05; lr: 0.00100; 16831/11084 tok/s;    262 sec\n",
            "[2020-04-20 21:15:27,205 INFO] Step 6800/ 7000; acc:  97.73; ppl:  1.13; xent: 0.13; lr: 0.00100; 16805/11201 tok/s;    264 sec\n",
            "[2020-04-20 21:15:29,026 INFO] Step 6850/ 7000; acc:  98.58; ppl:  1.08; xent: 0.08; lr: 0.00100; 15826/10730 tok/s;    266 sec\n",
            "[2020-04-20 21:15:30,958 INFO] Step 6900/ 7000; acc:  98.66; ppl:  1.06; xent: 0.06; lr: 0.00100; 17394/11517 tok/s;    268 sec\n",
            "[2020-04-20 21:15:33,004 INFO] Step 6950/ 7000; acc:  98.67; ppl:  1.06; xent: 0.06; lr: 0.00100; 17285/11609 tok/s;    270 sec\n",
            "[2020-04-20 21:15:34,689 INFO] Step 7000/ 7000; acc:  99.29; ppl:  1.03; xent: 0.03; lr: 0.00100; 16530/10835 tok/s;    271 sec\n",
            "[2020-04-20 21:15:34,692 INFO] Saving checkpoint trained-model_step_7000.pt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqGACHuWt52G",
        "colab_type": "text"
      },
      "source": [
        "![Lemmatizer model](https://github.com/TurkuNLP/Deep_Learning_in_LangTech_course/raw/master/figs/lemmatizer-model.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "osNuwMu-euZv",
        "colab_type": "code",
        "outputId": "9fa56c1d-ae66-493f-e6ab-e3ce10e068c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 421
        }
      },
      "source": [
        "# predict \n",
        "!cat dev.input | head -20 > small_test.input ; onmt_translate -model trained-model_step_7000.pt -src small_test.input -output pred.txt -replace_unk ; paste -d\"\\t\" small_test.input pred.txt"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2020-04-20 21:15:38,406 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0221, PRED PPL: 1.0223\n",
            "T h e PROPN Case=Nom Number=Sing\tT h e\n",
            "G a r d e n PROPN Case=Nom Number=Sing\tG a r d e n\n",
            "C o l l e c t i o n PROPN Case=Nom Number=Sing\tC o l l e c t i o n\n",
            "b y PROPN Case=Nom Number=Sing\tb y\n",
            "H & M PROPN Abbr=Yes Case=Nom Number=Sing\tH & M\n",
            "V i i k o n l o p u n NOUN Case=Gen Derivation=U Number=Sing\tv i i k o n # l o p p u\n",
            "p y ö r i t y s NOUN Case=Nom Number=Sing\tp y ö r i t y s\n",
            "a l k o i VERB Mood=Ind Number=Sing Person=3 Tense=Past VerbForm=Fin Voice=Act\ta l k a a\n",
            "H & M : n PROPN Abbr=Yes Case=Gen Number=Sing\tH & M\n",
            "j ä r j e s t ä m ä l l ä VERB Case=Ade Degree=Pos Number=Sing PartForm=Agt VerbForm=Part Voice=Act\tj ä r j e s t ä ä\n",
            "b l o g g a a j a b r u n s s i l l a NOUN Case=Ade Number=Sing\tb l o g g a a j a # b r u n s s i\n",
            "H e l s i n g i s s ä PROPN Case=Ine Number=Sing\tH e l s i n k i\n",
            ". PUNCT _\t.\n",
            "S h o w r o o m i l l a PROPN Case=Ade Number=Sing\tS h o w r o o m i\n",
            "e s i t e l t i i n VERB Mood=Ind Tense=Past VerbForm=Fin Voice=Pass\te s i t e l l ä\n",
            "u u s i ADJ Case=Nom Degree=Pos Number=Sing\tu u s i\n",
            "T h e PROPN Case=Nom Number=Sing\tT h e\n",
            "G a r d e n PROPN Case=Nom Number=Sing\tG a r d e n\n",
            "C o l l e c t i o n PROPN Case=Nom Number=Sing\tC o l l e c t i o n\n",
            "j a CCONJ _\tj a\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nHrW7jCBr9_u",
        "colab_type": "code",
        "outputId": "f716c94b-eaad-43ad-d440-706e063221b1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 127
        }
      },
      "source": [
        "# ambiguous words\n",
        "!echo \"k o i r a s t a NOUN Case=Ela Number=Sing\" > tmp.tmp ; onmt_translate -model trained-model_step_7000.pt -src tmp.tmp -output pred.txt -replace_unk ; cat pred.txt | perl -pe 's/ //g'\n",
        "!echo \"k o i r a s t a NOUN Case=Par Number=Sing\" > tmp.tmp ; onmt_translate -model trained-model_step_7000.pt -src tmp.tmp -output pred.txt -replace_unk ; cat pred.txt | perl -pe 's/ //g'"
      ],
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2020-04-20 21:15:42,854 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0007, PRED PPL: 1.0007\n",
            "koira\n",
            "[2020-04-20 21:15:47,637 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0073, PRED PPL: 1.0073\n",
            "koiras\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BlnhSki3P_Aq",
        "colab_type": "text"
      },
      "source": [
        "## Fully trained models available at https://turkunlp.org/Turku-neural-parser-pipeline/\n",
        "  - Includes trained models for segmentation, part-of-speech and morphological tagging, lemmatization, and syntactic parsing\n",
        "  - Over 50 languages supported\n",
        "  - Finnish lemmatization accuracy: ~95%\n",
        "  - Demo: http://bionlp-www.utu.fi/parser_demo/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KyxpvscB5OMM",
        "colab_type": "text"
      },
      "source": [
        "# Word inflection model\n",
        "- The model can also be trained 'the other way around'\n",
        "    - Generate the inflected word from the given lemma and desired inflection (for example Case information)\n",
        "    - (Very handy for a language learner! 😉)\n",
        "\n",
        "- We just need to modify the data preparation code\n",
        "  - Input: lemma + morphological features\n",
        "  - Output: inflected word\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WpgxiKgF74Z8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        },
        "outputId": "61480ef2-1ca7-4e73-a9e2-9b04fd38f55b"
      },
      "source": [
        "def prepare_inflection_dataset(fname):\n",
        "  # create input and output sequences (in text format)\n",
        "  data = []\n",
        "  with open(fname, \"rt\", encoding=\"utf-8\") as f:\n",
        "    for comments, sent in read_conllu(f):\n",
        "      for token in sent:\n",
        "        if \"-\" in token[ID] or \".\" in token[ID]: # multiword token or null node --> skip\n",
        "          continue\n",
        "        if token[UPOS] != \"NOUN\": # let's take only nouns\n",
        "          continue\n",
        "        input_chars = \" \".join(c for c in token[LEMMA]) # input is lemma + morphological features\n",
        "        features = \" \".join([token[UPOS]] + token[FEATS].split(\"|\")) # add morphological features\n",
        "        input_chars = input_chars + \" \" + features\n",
        "        output_chars = \" \".join(c for c in token[FORM])\n",
        "        data.append((input_chars, output_chars))\n",
        "  return data\n",
        "\n",
        "\n",
        "# transform data\n",
        "train_data = prepare_inflection_dataset(\"fi_tdt-ud-train.conllu\")\n",
        "import random\n",
        "random.shuffle(train_data)\n",
        "\n",
        "dev_data = prepare_inflection_dataset(\"fi_tdt-ud-dev.conllu\")\n",
        "print(\"First train examples:\", train_data[:5])\n",
        "print(\"Number of training examples:\", len(train_data))\n",
        "print(\"\\nNumber of development examples:\", len(dev_data))\n",
        "\n",
        "save_data(train_data, dev_data)"
      ],
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First train examples: [('l a t v a # j ä r v i NOUN Case=Par Number=Plur', 'l a t v a j ä r v i ä'), ('a l u s NOUN Case=Ine Number=Plur', 'a l u k s i s s a'), ('c u r r y NOUN Case=Par Number=Sing', 'c u r r y a'), ('k i i n n o s t u s NOUN Case=Gen Number=Sing', 'K i i n n o s t u k s e n'), ('h a k u # k o n e # y h t i ö NOUN Case=Nom Number=Plur', 'H a k u k o n e y h t i ö t')]\n",
            "Number of training examples: 45588\n",
            "\n",
            "Number of development examples: 5103\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CQvbPI3nH89O",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "7cbc675e-8129-48d4-97b6-e3433b4fefee"
      },
      "source": [
        "# run preprocessing, this handles vectorization etc.\n",
        "!onmt_preprocess -train_src train.input -train_tgt train.output -valid_src dev.input -valid_tgt dev.output -save_data preprocessed-data -src_words_min_frequency 5 -tgt_words_min_frequency 5 -overwrite\n",
        "\n",
        "# train model\n",
        "\n",
        "BATCH_SIZE = 64\n",
        "LEARNING_RATE = 0.001\n",
        "OPTIMIZER = \"adam\"\n",
        "ENCODER = \"brnn\"\n",
        "TRAIN_STEPS = 5000 #step is one minibatch, so train for 5000 minibatches\n",
        "\n",
        "print(\"How many epochs:\", (TRAIN_STEPS*BATCH_SIZE)/len(train_data))\n",
        "\n",
        "!onmt_train -data preprocessed-data -save_model trained-model -gpu_ranks 0 -learning_rate {LEARNING_RATE} -batch_size {BATCH_SIZE} -optim {OPTIMIZER} -train_steps {TRAIN_STEPS} -save_checkpoint_steps {TRAIN_STEPS} -encoder_type {ENCODER}"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2020-04-20 21:15:52,682 INFO] Extracting features...\n",
            "[2020-04-20 21:15:52,683 INFO]  * number of source features: 0.\n",
            "[2020-04-20 21:15:52,683 INFO]  * number of target features: 0.\n",
            "[2020-04-20 21:15:52,683 INFO] Building `Fields` object...\n",
            "[2020-04-20 21:15:52,683 INFO] Building & saving training data...\n",
            "[2020-04-20 21:15:52,684 WARNING] Shards for corpus train already exist, will be overwritten because `-overwrite` option is set.\n",
            "[2020-04-20 21:15:52,690 WARNING] Overwrite shards for corpus None\n",
            "[2020-04-20 21:15:52,737 INFO] Building shard 0.\n",
            "[2020-04-20 21:15:53,991 INFO]  * saving 0th train data shard to preprocessed-data.train.0.pt.\n",
            "[2020-04-20 21:15:54,897 INFO]  * tgt vocab size: 76.\n",
            "[2020-04-20 21:15:54,897 INFO]  * src vocab size: 112.\n",
            "[2020-04-20 21:15:54,899 INFO] Building & saving validation data...\n",
            "[2020-04-20 21:15:54,900 WARNING] Shards for corpus valid already exist, will be overwritten because `-overwrite` option is set.\n",
            "[2020-04-20 21:15:54,904 WARNING] Overwrite shards for corpus None\n",
            "[2020-04-20 21:15:54,916 INFO] Building shard 0.\n",
            "[2020-04-20 21:15:55,013 INFO]  * saving 0th valid data shard to preprocessed-data.valid.0.pt.\n",
            "How many epochs: 7.019391067824866\n",
            "[2020-04-20 21:15:58,244 INFO]  * src vocab size = 112\n",
            "[2020-04-20 21:15:58,244 INFO]  * tgt vocab size = 76\n",
            "[2020-04-20 21:15:58,244 INFO] Building model...\n",
            "[2020-04-20 21:16:01,473 INFO] NMTModel(\n",
            "  (encoder): RNNEncoder(\n",
            "    (embeddings): Embeddings(\n",
            "      (make_embedding): Sequential(\n",
            "        (emb_luts): Elementwise(\n",
            "          (0): Embedding(112, 500, padding_idx=1)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (rnn): LSTM(500, 250, num_layers=2, dropout=0.3, bidirectional=True)\n",
            "  )\n",
            "  (decoder): InputFeedRNNDecoder(\n",
            "    (embeddings): Embeddings(\n",
            "      (make_embedding): Sequential(\n",
            "        (emb_luts): Elementwise(\n",
            "          (0): Embedding(76, 500, padding_idx=1)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (dropout): Dropout(p=0.3, inplace=False)\n",
            "    (rnn): StackedLSTM(\n",
            "      (dropout): Dropout(p=0.3, inplace=False)\n",
            "      (layers): ModuleList(\n",
            "        (0): LSTMCell(1000, 500)\n",
            "        (1): LSTMCell(500, 500)\n",
            "      )\n",
            "    )\n",
            "    (attn): GlobalAttention(\n",
            "      (linear_in): Linear(in_features=500, out_features=500, bias=False)\n",
            "      (linear_out): Linear(in_features=1000, out_features=500, bias=False)\n",
            "    )\n",
            "  )\n",
            "  (generator): Sequential(\n",
            "    (0): Linear(in_features=500, out_features=76, bias=True)\n",
            "    (1): Cast()\n",
            "    (2): LogSoftmax()\n",
            "  )\n",
            ")\n",
            "[2020-04-20 21:16:01,474 INFO] encoder: 3064000\n",
            "[2020-04-20 21:16:01,474 INFO] decoder: 5834076\n",
            "[2020-04-20 21:16:01,474 INFO] * number of parameters: 8898076\n",
            "[2020-04-20 21:16:01,475 INFO] Starting training on GPU: [0]\n",
            "[2020-04-20 21:16:01,475 INFO] Start training loop and validate every 10000 steps...\n",
            "[2020-04-20 21:16:01,475 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:16:01,758 INFO] number of examples: 45588\n",
            "[2020-04-20 21:16:04,522 INFO] Step 50/ 5000; acc:  16.45; ppl: 18.94; xent: 2.94; lr: 0.00100; 12036/11474 tok/s;      3 sec\n",
            "[2020-04-20 21:16:07,398 INFO] Step 100/ 5000; acc:  36.36; ppl:  8.36; xent: 2.12; lr: 0.00100; 14184/13287 tok/s;      6 sec\n",
            "[2020-04-20 21:16:09,940 INFO] Step 150/ 5000; acc:  59.93; ppl:  4.02; xent: 1.39; lr: 0.00100; 14634/13659 tok/s;      8 sec\n",
            "[2020-04-20 21:16:12,383 INFO] Step 200/ 5000; acc:  74.87; ppl:  2.42; xent: 0.88; lr: 0.00100; 14563/13474 tok/s;     11 sec\n",
            "[2020-04-20 21:16:14,639 INFO] Step 250/ 5000; acc:  79.48; ppl:  2.16; xent: 0.77; lr: 0.00100; 14583/13153 tok/s;     13 sec\n",
            "[2020-04-20 21:16:17,147 INFO] Step 300/ 5000; acc:  84.04; ppl:  1.87; xent: 0.63; lr: 0.00100; 14399/13462 tok/s;     16 sec\n",
            "[2020-04-20 21:16:19,751 INFO] Step 350/ 5000; acc:  86.91; ppl:  1.67; xent: 0.51; lr: 0.00100; 14444/13679 tok/s;     18 sec\n",
            "[2020-04-20 21:16:22,330 INFO] Step 400/ 5000; acc:  86.79; ppl:  1.86; xent: 0.62; lr: 0.00100; 14461/13408 tok/s;     21 sec\n",
            "[2020-04-20 21:16:25,028 INFO] Step 450/ 5000; acc:  91.69; ppl:  1.41; xent: 0.35; lr: 0.00100; 14798/13644 tok/s;     24 sec\n",
            "[2020-04-20 21:16:27,627 INFO] Step 500/ 5000; acc:  91.75; ppl:  1.42; xent: 0.35; lr: 0.00100; 14219/13482 tok/s;     26 sec\n",
            "[2020-04-20 21:16:30,424 INFO] Step 550/ 5000; acc:  93.82; ppl:  1.29; xent: 0.25; lr: 0.00100; 14654/13969 tok/s;     29 sec\n",
            "[2020-04-20 21:16:32,919 INFO] Step 600/ 5000; acc:  94.14; ppl:  1.30; xent: 0.27; lr: 0.00100; 14358/13678 tok/s;     31 sec\n",
            "[2020-04-20 21:16:35,215 INFO] Step 650/ 5000; acc:  95.17; ppl:  1.23; xent: 0.20; lr: 0.00100; 14226/13200 tok/s;     34 sec\n",
            "[2020-04-20 21:16:37,769 INFO] Step 700/ 5000; acc:  95.83; ppl:  1.18; xent: 0.17; lr: 0.00100; 14379/13700 tok/s;     36 sec\n",
            "[2020-04-20 21:16:38,392 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:16:38,830 INFO] number of examples: 45588\n",
            "[2020-04-20 21:16:40,920 INFO] Step 750/ 5000; acc:  96.35; ppl:  1.18; xent: 0.17; lr: 0.00100; 11758/11157 tok/s;     39 sec\n",
            "[2020-04-20 21:16:43,613 INFO] Step 800/ 5000; acc:  96.06; ppl:  1.20; xent: 0.18; lr: 0.00100; 14171/13414 tok/s;     42 sec\n",
            "[2020-04-20 21:16:46,286 INFO] Step 850/ 5000; acc:  96.35; ppl:  1.18; xent: 0.17; lr: 0.00100; 14921/13734 tok/s;     45 sec\n",
            "[2020-04-20 21:16:48,656 INFO] Step 900/ 5000; acc:  96.41; ppl:  1.15; xent: 0.14; lr: 0.00100; 14423/13417 tok/s;     47 sec\n",
            "[2020-04-20 21:16:51,060 INFO] Step 950/ 5000; acc:  96.69; ppl:  1.14; xent: 0.13; lr: 0.00100; 14404/13380 tok/s;     50 sec\n",
            "[2020-04-20 21:16:53,389 INFO] Step 1000/ 5000; acc:  96.12; ppl:  1.20; xent: 0.18; lr: 0.00100; 14282/13210 tok/s;     52 sec\n",
            "[2020-04-20 21:16:56,099 INFO] Step 1050/ 5000; acc:  96.55; ppl:  1.16; xent: 0.15; lr: 0.00100; 14767/13782 tok/s;     55 sec\n",
            "[2020-04-20 21:16:58,685 INFO] Step 1100/ 5000; acc:  92.98; ppl:  1.53; xent: 0.42; lr: 0.00100; 14248/13450 tok/s;     57 sec\n",
            "[2020-04-20 21:17:01,279 INFO] Step 1150/ 5000; acc:  97.12; ppl:  1.13; xent: 0.13; lr: 0.00100; 14701/13494 tok/s;     60 sec\n",
            "[2020-04-20 21:17:03,908 INFO] Step 1200/ 5000; acc:  96.98; ppl:  1.15; xent: 0.14; lr: 0.00100; 14426/13572 tok/s;     62 sec\n",
            "[2020-04-20 21:17:06,658 INFO] Step 1250/ 5000; acc:  96.71; ppl:  1.17; xent: 0.15; lr: 0.00100; 14476/13735 tok/s;     65 sec\n",
            "[2020-04-20 21:17:09,147 INFO] Step 1300/ 5000; acc:  97.85; ppl:  1.09; xent: 0.09; lr: 0.00100; 14195/13708 tok/s;     68 sec\n",
            "[2020-04-20 21:17:11,572 INFO] Step 1350/ 5000; acc:  97.33; ppl:  1.11; xent: 0.11; lr: 0.00100; 14245/13260 tok/s;     70 sec\n",
            "[2020-04-20 21:17:14,143 INFO] Step 1400/ 5000; acc:  97.87; ppl:  1.10; xent: 0.10; lr: 0.00100; 14268/13537 tok/s;     73 sec\n",
            "[2020-04-20 21:17:15,412 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:17:15,758 INFO] number of examples: 45588\n",
            "[2020-04-20 21:17:17,201 INFO] Step 1450/ 5000; acc:  97.95; ppl:  1.09; xent: 0.09; lr: 0.00100; 12160/11407 tok/s;     76 sec\n",
            "[2020-04-20 21:17:19,873 INFO] Step 1500/ 5000; acc:  97.55; ppl:  1.12; xent: 0.11; lr: 0.00100; 13947/13255 tok/s;     78 sec\n",
            "[2020-04-20 21:17:22,494 INFO] Step 1550/ 5000; acc:  97.58; ppl:  1.11; xent: 0.11; lr: 0.00100; 14827/13748 tok/s;     81 sec\n",
            "[2020-04-20 21:17:24,955 INFO] Step 1600/ 5000; acc:  97.96; ppl:  1.09; xent: 0.08; lr: 0.00100; 14540/13532 tok/s;     83 sec\n",
            "[2020-04-20 21:17:27,382 INFO] Step 1650/ 5000; acc:  97.67; ppl:  1.10; xent: 0.10; lr: 0.00100; 14399/13468 tok/s;     86 sec\n",
            "[2020-04-20 21:17:29,688 INFO] Step 1700/ 5000; acc:  97.53; ppl:  1.11; xent: 0.10; lr: 0.00100; 14425/13104 tok/s;     88 sec\n",
            "[2020-04-20 21:17:32,348 INFO] Step 1750/ 5000; acc:  97.75; ppl:  1.11; xent: 0.10; lr: 0.00100; 14593/13644 tok/s;     91 sec\n",
            "[2020-04-20 21:17:35,069 INFO] Step 1800/ 5000; acc:  96.98; ppl:  1.15; xent: 0.14; lr: 0.00100; 14426/13546 tok/s;     94 sec\n",
            "[2020-04-20 21:17:37,587 INFO] Step 1850/ 5000; acc:  97.19; ppl:  1.14; xent: 0.13; lr: 0.00100; 14702/13566 tok/s;     96 sec\n",
            "[2020-04-20 21:17:40,203 INFO] Step 1900/ 5000; acc:  97.84; ppl:  1.10; xent: 0.09; lr: 0.00100; 14456/13612 tok/s;     99 sec\n",
            "[2020-04-20 21:17:42,889 INFO] Step 1950/ 5000; acc:  97.83; ppl:  1.10; xent: 0.09; lr: 0.00100; 14323/13460 tok/s;    101 sec\n",
            "[2020-04-20 21:17:45,496 INFO] Step 2000/ 5000; acc:  98.02; ppl:  1.09; xent: 0.08; lr: 0.00100; 14413/13775 tok/s;    104 sec\n",
            "[2020-04-20 21:17:47,974 INFO] Step 2050/ 5000; acc:  97.77; ppl:  1.09; xent: 0.09; lr: 0.00100; 14147/13433 tok/s;    106 sec\n",
            "[2020-04-20 21:17:50,411 INFO] Step 2100/ 5000; acc:  98.03; ppl:  1.09; xent: 0.09; lr: 0.00100; 14160/13485 tok/s;    109 sec\n",
            "[2020-04-20 21:17:52,376 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:17:52,756 INFO] number of examples: 45588\n",
            "[2020-04-20 21:17:53,550 INFO] Step 2150/ 5000; acc:  98.31; ppl:  1.07; xent: 0.07; lr: 0.00100; 12072/11393 tok/s;    112 sec\n",
            "[2020-04-20 21:17:56,151 INFO] Step 2200/ 5000; acc:  97.89; ppl:  1.10; xent: 0.10; lr: 0.00100; 14052/13152 tok/s;    115 sec\n",
            "[2020-04-20 21:17:58,826 INFO] Step 2250/ 5000; acc:  97.72; ppl:  1.12; xent: 0.12; lr: 0.00100; 14530/13819 tok/s;    117 sec\n",
            "[2020-04-20 21:18:01,403 INFO] Step 2300/ 5000; acc:  98.28; ppl:  1.08; xent: 0.07; lr: 0.00100; 14775/13591 tok/s;    120 sec\n",
            "[2020-04-20 21:18:03,785 INFO] Step 2350/ 5000; acc:  97.99; ppl:  1.09; xent: 0.09; lr: 0.00100; 14350/13339 tok/s;    122 sec\n",
            "[2020-04-20 21:18:06,072 INFO] Step 2400/ 5000; acc:  97.94; ppl:  1.09; xent: 0.08; lr: 0.00100; 14128/13018 tok/s;    125 sec\n",
            "[2020-04-20 21:18:08,681 INFO] Step 2450/ 5000; acc:  98.00; ppl:  1.09; xent: 0.09; lr: 0.00100; 14731/13570 tok/s;    127 sec\n",
            "[2020-04-20 21:18:11,315 INFO] Step 2500/ 5000; acc:  97.65; ppl:  1.11; xent: 0.11; lr: 0.00100; 14403/13579 tok/s;    130 sec\n",
            "[2020-04-20 21:18:13,949 INFO] Step 2550/ 5000; acc:  95.99; ppl:  1.26; xent: 0.23; lr: 0.00100; 14570/13542 tok/s;    132 sec\n",
            "[2020-04-20 21:18:16,592 INFO] Step 2600/ 5000; acc:  98.01; ppl:  1.10; xent: 0.09; lr: 0.00100; 14586/13597 tok/s;    135 sec\n",
            "[2020-04-20 21:18:19,134 INFO] Step 2650/ 5000; acc:  98.06; ppl:  1.09; xent: 0.08; lr: 0.00100; 14069/13234 tok/s;    138 sec\n",
            "[2020-04-20 21:18:21,945 INFO] Step 2700/ 5000; acc:  97.96; ppl:  1.10; xent: 0.09; lr: 0.00100; 14514/13897 tok/s;    140 sec\n",
            "[2020-04-20 21:18:24,400 INFO] Step 2750/ 5000; acc:  98.01; ppl:  1.08; xent: 0.08; lr: 0.00100; 14254/13515 tok/s;    143 sec\n",
            "[2020-04-20 21:18:26,838 INFO] Step 2800/ 5000; acc:  98.01; ppl:  1.09; xent: 0.09; lr: 0.00100; 14305/13212 tok/s;    145 sec\n",
            "[2020-04-20 21:18:29,353 INFO] Step 2850/ 5000; acc:  98.29; ppl:  1.07; xent: 0.07; lr: 0.00100; 14176/13676 tok/s;    148 sec\n",
            "[2020-04-20 21:18:29,439 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:18:29,810 INFO] number of examples: 45588\n",
            "[2020-04-20 21:18:32,445 INFO] Step 2900/ 5000; acc:  98.47; ppl:  1.06; xent: 0.06; lr: 0.00100; 11986/11416 tok/s;    151 sec\n",
            "[2020-04-20 21:18:35,234 INFO] Step 2950/ 5000; acc:  97.91; ppl:  1.11; xent: 0.10; lr: 0.00100; 14283/13399 tok/s;    154 sec\n",
            "[2020-04-20 21:18:37,812 INFO] Step 3000/ 5000; acc:  98.37; ppl:  1.07; xent: 0.07; lr: 0.00100; 14775/13789 tok/s;    156 sec\n",
            "[2020-04-20 21:18:40,224 INFO] Step 3050/ 5000; acc:  98.31; ppl:  1.07; xent: 0.07; lr: 0.00100; 14592/13488 tok/s;    159 sec\n",
            "[2020-04-20 21:18:42,441 INFO] Step 3100/ 5000; acc:  98.30; ppl:  1.07; xent: 0.06; lr: 0.00100; 14525/13129 tok/s;    161 sec\n",
            "[2020-04-20 21:18:44,961 INFO] Step 3150/ 5000; acc:  98.22; ppl:  1.08; xent: 0.07; lr: 0.00100; 14481/13472 tok/s;    163 sec\n",
            "[2020-04-20 21:18:47,592 INFO] Step 3200/ 5000; acc:  98.21; ppl:  1.08; xent: 0.08; lr: 0.00100; 14513/13780 tok/s;    166 sec\n",
            "[2020-04-20 21:18:50,175 INFO] Step 3250/ 5000; acc:  98.01; ppl:  1.08; xent: 0.08; lr: 0.00100; 14490/13464 tok/s;    169 sec\n",
            "[2020-04-20 21:18:52,858 INFO] Step 3300/ 5000; acc:  98.38; ppl:  1.07; xent: 0.07; lr: 0.00100; 14666/13553 tok/s;    171 sec\n",
            "[2020-04-20 21:18:55,463 INFO] Step 3350/ 5000; acc:  98.22; ppl:  1.08; xent: 0.08; lr: 0.00100; 14260/13490 tok/s;    174 sec\n",
            "[2020-04-20 21:18:58,240 INFO] Step 3400/ 5000; acc:  98.42; ppl:  1.07; xent: 0.06; lr: 0.00100; 14641/13984 tok/s;    177 sec\n",
            "[2020-04-20 21:19:00,733 INFO] Step 3450/ 5000; acc:  98.48; ppl:  1.07; xent: 0.06; lr: 0.00100; 14372/13563 tok/s;    179 sec\n",
            "[2020-04-20 21:19:03,033 INFO] Step 3500/ 5000; acc:  98.32; ppl:  1.06; xent: 0.06; lr: 0.00100; 14190/13225 tok/s;    182 sec\n",
            "[2020-04-20 21:19:05,602 INFO] Step 3550/ 5000; acc:  98.24; ppl:  1.07; xent: 0.07; lr: 0.00100; 14254/13569 tok/s;    184 sec\n",
            "[2020-04-20 21:19:06,344 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:19:06,707 INFO] number of examples: 45588\n",
            "[2020-04-20 21:19:08,678 INFO] Step 3600/ 5000; acc:  98.55; ppl:  1.06; xent: 0.06; lr: 0.00100; 11965/11377 tok/s;    187 sec\n",
            "[2020-04-20 21:19:11,416 INFO] Step 3650/ 5000; acc:  98.32; ppl:  1.07; xent: 0.07; lr: 0.00100; 14288/13450 tok/s;    190 sec\n",
            "[2020-04-20 21:19:14,020 INFO] Step 3700/ 5000; acc:  98.27; ppl:  1.08; xent: 0.08; lr: 0.00100; 14872/13776 tok/s;    193 sec\n",
            "[2020-04-20 21:19:16,419 INFO] Step 3750/ 5000; acc:  98.26; ppl:  1.07; xent: 0.07; lr: 0.00100; 14461/13387 tok/s;    195 sec\n",
            "[2020-04-20 21:19:18,851 INFO] Step 3800/ 5000; acc:  98.50; ppl:  1.06; xent: 0.06; lr: 0.00100; 14474/13410 tok/s;    197 sec\n",
            "[2020-04-20 21:19:21,145 INFO] Step 3850/ 5000; acc:  98.23; ppl:  1.07; xent: 0.07; lr: 0.00100; 14256/13223 tok/s;    200 sec\n",
            "[2020-04-20 21:19:23,829 INFO] Step 3900/ 5000; acc:  98.20; ppl:  1.08; xent: 0.08; lr: 0.00100; 14621/13645 tok/s;    202 sec\n",
            "[2020-04-20 21:19:26,457 INFO] Step 3950/ 5000; acc:  97.78; ppl:  1.11; xent: 0.11; lr: 0.00100; 14289/13479 tok/s;    205 sec\n",
            "[2020-04-20 21:19:29,063 INFO] Step 4000/ 5000; acc:  98.03; ppl:  1.09; xent: 0.09; lr: 0.00100; 14814/13602 tok/s;    208 sec\n",
            "[2020-04-20 21:19:31,710 INFO] Step 4050/ 5000; acc:  97.99; ppl:  1.10; xent: 0.09; lr: 0.00100; 14343/13489 tok/s;    210 sec\n",
            "[2020-04-20 21:19:34,437 INFO] Step 4100/ 5000; acc:  98.31; ppl:  1.07; xent: 0.07; lr: 0.00100; 14576/13805 tok/s;    213 sec\n",
            "[2020-04-20 21:19:36,924 INFO] Step 4150/ 5000; acc:  98.58; ppl:  1.06; xent: 0.06; lr: 0.00100; 14180/13692 tok/s;    215 sec\n",
            "[2020-04-20 21:19:39,358 INFO] Step 4200/ 5000; acc:  98.15; ppl:  1.07; xent: 0.07; lr: 0.00100; 14243/13288 tok/s;    218 sec\n",
            "[2020-04-20 21:19:41,903 INFO] Step 4250/ 5000; acc:  98.43; ppl:  1.07; xent: 0.07; lr: 0.00100; 14315/13627 tok/s;    220 sec\n",
            "[2020-04-20 21:19:43,282 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:19:43,746 INFO] number of examples: 45588\n",
            "[2020-04-20 21:19:45,099 INFO] Step 4300/ 5000; acc:  98.53; ppl:  1.06; xent: 0.06; lr: 0.00100; 11695/10994 tok/s;    224 sec\n",
            "[2020-04-20 21:19:47,747 INFO] Step 4350/ 5000; acc:  98.34; ppl:  1.07; xent: 0.07; lr: 0.00100; 13973/13188 tok/s;    226 sec\n",
            "[2020-04-20 21:19:50,375 INFO] Step 4400/ 5000; acc:  98.28; ppl:  1.08; xent: 0.08; lr: 0.00100; 14763/13799 tok/s;    229 sec\n",
            "[2020-04-20 21:19:52,845 INFO] Step 4450/ 5000; acc:  98.50; ppl:  1.06; xent: 0.06; lr: 0.00100; 14570/13565 tok/s;    231 sec\n",
            "[2020-04-20 21:19:55,232 INFO] Step 4500/ 5000; acc:  98.49; ppl:  1.06; xent: 0.06; lr: 0.00100; 14531/13504 tok/s;    234 sec\n",
            "[2020-04-20 21:19:57,563 INFO] Step 4550/ 5000; acc:  98.23; ppl:  1.07; xent: 0.06; lr: 0.00100; 14406/13072 tok/s;    236 sec\n",
            "[2020-04-20 21:20:00,222 INFO] Step 4600/ 5000; acc:  98.42; ppl:  1.07; xent: 0.07; lr: 0.00100; 14673/13771 tok/s;    239 sec\n",
            "[2020-04-20 21:20:02,908 INFO] Step 4650/ 5000; acc:  98.03; ppl:  1.09; xent: 0.08; lr: 0.00100; 14422/13578 tok/s;    241 sec\n",
            "[2020-04-20 21:20:05,441 INFO] Step 4700/ 5000; acc:  98.40; ppl:  1.06; xent: 0.06; lr: 0.00100; 14768/13563 tok/s;    244 sec\n",
            "[2020-04-20 21:20:08,063 INFO] Step 4750/ 5000; acc:  98.47; ppl:  1.07; xent: 0.06; lr: 0.00100; 14475/13623 tok/s;    247 sec\n",
            "[2020-04-20 21:20:10,692 INFO] Step 4800/ 5000; acc:  98.23; ppl:  1.08; xent: 0.07; lr: 0.00100; 14296/13473 tok/s;    249 sec\n",
            "[2020-04-20 21:20:13,346 INFO] Step 4850/ 5000; acc:  98.66; ppl:  1.06; xent: 0.06; lr: 0.00100; 14537/13838 tok/s;    252 sec\n",
            "[2020-04-20 21:20:15,796 INFO] Step 4900/ 5000; acc:  98.31; ppl:  1.07; xent: 0.06; lr: 0.00100; 14255/13533 tok/s;    254 sec\n",
            "[2020-04-20 21:20:18,203 INFO] Step 4950/ 5000; acc:  98.44; ppl:  1.06; xent: 0.06; lr: 0.00100; 14202/13439 tok/s;    257 sec\n",
            "[2020-04-20 21:20:20,276 INFO] Loading dataset from preprocessed-data.train.0.pt\n",
            "[2020-04-20 21:20:20,656 INFO] number of examples: 45588\n",
            "[2020-04-20 21:20:21,356 INFO] Step 5000/ 5000; acc:  98.63; ppl:  1.06; xent: 0.05; lr: 0.00100; 12080/11479 tok/s;    260 sec\n",
            "[2020-04-20 21:20:21,357 INFO] Saving checkpoint trained-model_step_5000.pt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VjCIAXNzSTl2",
        "colab_type": "text"
      },
      "source": [
        "## Predict Inflections"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-589URYqSPee",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        },
        "outputId": "451853c2-c1c8-4903-87b6-a8cdabe85b40"
      },
      "source": [
        "!echo \"k i s s a NOUN Case=Ade Number=Sing\" > tmp.tmp ; onmt_translate -model trained-model_step_5000.pt -src tmp.tmp -output pred.txt ; cat pred.txt | perl -pe 's/ //g'\n",
        "!echo \"t a l o NOUN Case=Ine Number=Sing\" > tmp.tmp ; onmt_translate -model trained-model_step_5000.pt -src tmp.tmp -output pred.txt ; cat pred.txt | perl -pe 's/ //g'\n",
        "!echo \"t u l p p a a n i NOUN Case=Par Number=Plur\" > tmp.tmp ; onmt_translate -model trained-model_step_5000.pt -src tmp.tmp -output pred.txt ; cat pred.txt | perl -pe 's/ //g'\n",
        "!echo \"t u l p p a a n i NOUN Case=Par Clitic=Kin Number=Plur\" > tmp.tmp ; onmt_translate -model trained-model_step_5000.pt -src tmp.tmp -output pred.txt ; cat pred.txt | perl -pe 's/ //g'"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2020-04-20 21:20:25,975 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0068, PRED PPL: 1.0069\n",
            "kissalla\n",
            "[2020-04-20 21:20:29,071 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0034, PRED PPL: 1.0034\n",
            "talossa\n",
            "[2020-04-20 21:20:33,170 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0124, PRED PPL: 1.0124\n",
            "tulppaaneja\n",
            "[2020-04-20 21:20:37,771 INFO] Translating shard 0.\n",
            "PRED AVG SCORE: -0.0315, PRED PPL: 1.0320\n",
            "tulppaanejakin\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}