light_weight_roberta_base_baseline.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "light-weight-roberta-base-baseline.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nishipy/clrp/blob/main/light_weight_roberta_base_baseline.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M5tA-mkxXV5b"
      },
      "source": [
        "# Overview\n",
        "This is kernel is almost the same as [Lightweight Roberta solution in PyTorch](https://www.kaggle.com/andretugan/lightweight-roberta-solution-in-pytorch), but instead of \"roberta-base\", it starts from [Maunish's pre-trained model](https://www.kaggle.com/maunish/clrp-roberta-base).\n",
        "\n",
        "Acknowledgments: some ideas were taken from kernels by [Torch](https://www.kaggle.com/rhtsingh) and [Maunish](https://www.kaggle.com/maunish).\n",
        "\n",
        "In addition, we use the [stratified_kfold train dataset](https://www.kaggle.com/takeshikobayashi/commonlit-train-datasetfor) training the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4H_YG8xbXZGN"
      },
      "source": [
        "## Original notebook\n",
        "- Lightweight Roberta solution\n",
        "  - https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch\n",
        "- pretraied with MLM\n",
        "  - https://www.kaggle.com/maunish/clrp-pytorch-roberta-pretrain"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBSvZK-MXsRq"
      },
      "source": [
        "# Prepare"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UtTPu3rhX22a"
      },
      "source": [
        "## Checking GPU status"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m4-zjkgTXyB7",
        "outputId": "3a07dcde-0487-4910-e65a-f931c01df5e8"
      },
      "source": [
        "gpu_info = !nvidia-smi\n",
        "gpu_info = '\\n'.join(gpu_info)\n",
        "if gpu_info.find('failed') >= 0:\n",
        "  print('Select the Runtime > \"Change runtime type\" menu to enable a GPU accelerator, ')\n",
        "  print('and then re-execute this cell.')\n",
        "else:\n",
        "  print(gpu_info)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sun Jul  4 06:53:44 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 465.27       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   40C    P0    33W / 250W |   7491MiB / 16280MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9niGsYeuYEgf"
      },
      "source": [
        "## Download dataset from kaggle"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1JKMEPT1YBzO",
        "outputId": "4107df47-7465-49c4-bf88-5121c8b7533e"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "12zbNycNavJc"
      },
      "source": [
        "### kaggle.json"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VaMQCChEYIiJ"
      },
      "source": [
        "!mkdir -p /root/.kaggle/\n",
        "!cp ./drive/MyDrive/kaggle/commonlit/kaggle.json ~/.kaggle/kaggle.json\n",
        "!chmod 600 ~/.kaggle/kaggle.json"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LampuH0tawt4"
      },
      "source": [
        "### Competition dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2Pn015V0Yamu",
        "outputId": "e9f152ef-b5b1-42e8-8bfb-a2102fbd6181"
      },
      "source": [
        "!mkdir -p ../input/commonlitreadabilityprize/\n",
        "!kaggle competitions download -c commonlitreadabilityprize -p ../input/commonlitreadabilityprize/\n",
        "!cp -f ./drive/MyDrive/kaggle/commonlit/train_stratiKfold.csv.zip ../input/commonlitreadabilityprize/"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.12 / client 1.5.4)\n",
            "Downloading train.csv.zip to ../input/commonlitreadabilityprize\n",
            "  0% 0.00/1.13M [00:00<?, ?B/s]\n",
            "100% 1.13M/1.13M [00:00<00:00, 78.2MB/s]\n",
            "Downloading sample_submission.csv to ../input/commonlitreadabilityprize\n",
            "  0% 0.00/108 [00:00<?, ?B/s]\n",
            "100% 108/108 [00:00<00:00, 91.7kB/s]\n",
            "Downloading test.csv to ../input/commonlitreadabilityprize\n",
            "  0% 0.00/6.79k [00:00<?, ?B/s]\n",
            "100% 6.79k/6.79k [00:00<00:00, 6.90MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_SouT_7pYuvb",
        "outputId": "1357c22e-0206-4037-afac-1f9ade013ceb"
      },
      "source": [
        "!unzip -o ../input/commonlitreadabilityprize/train.csv.zip -d ../input/commonlitreadabilityprize/\n",
        "!unzip -o ../input/commonlitreadabilityprize/train_stratiKfold.csv.zip -d ../input/commonlitreadabilityprize/"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  ../input/commonlitreadabilityprize/train.csv.zip\n",
            "  inflating: ../input/commonlitreadabilityprize/train.csv  \n",
            "Archive:  ../input/commonlitreadabilityprize/train_stratiKfold.csv.zip\n",
            "  inflating: ../input/commonlitreadabilityprize/train_stratiKfold.csv  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gjihJSBzZVt8",
        "outputId": "47b967ac-4aeb-4794-8012-086ef860ccdc"
      },
      "source": [
        "!ls ../input/commonlitreadabilityprize/"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "sample_submission.csv  train.csv      train_stratiKfold.csv\n",
            "test.csv\t       train.csv.zip  train_stratiKfold.csv.zip\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wujOl7iGa2-C"
      },
      "source": [
        "### Model pre-trained with MLM \n",
        "- Notebook\n",
        "  - https://www.kaggle.com/maunish/clrp-pytorch-roberta-pretrain\n",
        "- Model data\n",
        "  - https://www.kaggle.com/maunish/clrp-roberta-base"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s9EgsJFaa7UW",
        "outputId": "aef839dd-18e3-4eab-d045-8e125e3b0b07"
      },
      "source": [
        "!mkdir -p ../input/commonlitreadabilityprize/pretrained-model/\n",
        "!kaggle datasets download maunish/clrp-roberta-base -p ../input/commonlitreadabilityprize/pretrained-model/"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading clrp-roberta-base.zip to ../input/commonlitreadabilityprize/pretrained-model\n",
            "100% 3.00G/3.01G [00:25<00:00, 111MB/s]\n",
            "100% 3.01G/3.01G [00:25<00:00, 128MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9KA2Tzd5bfnJ",
        "outputId": "c14df241-47df-4e25-aba9-95eaaa5966cc"
      },
      "source": [
        "!unzip -o ../input/commonlitreadabilityprize/pretrained-model/clrp-roberta-base.zip -d ../input/commonlitreadabilityprize/pretrained-model/"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  ../input/commonlitreadabilityprize/pretrained-model/clrp-roberta-base.zip\n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/config.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/merges.txt  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/pytorch_model.bin  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/special_tokens_map.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/tokenizer_config.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/training_args.bin  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/vocab.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-600/config.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-600/optimizer.pt  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-600/pytorch_model.bin  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-600/scheduler.pt  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-600/trainer_state.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-600/training_args.bin  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-800/config.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-800/optimizer.pt  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-800/pytorch_model.bin  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-800/scheduler.pt  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-800/trainer_state.json  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base_chk/checkpoint-800/training_args.bin  \n",
            "  inflating: ../input/commonlitreadabilityprize/pretrained-model/text.txt  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oeCpZ6DEYH90"
      },
      "source": [
        "# Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BOrw8yaddJV0",
        "outputId": "84f277eb-d2c4-4b16-e93b-bd5f25482245"
      },
      "source": [
        "!pip install transformers accelerate datasets"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fd/1a/41c644c963249fd7f3836d926afa1e3f1cc234a1c40d80c5f03ad8f6f1b2/transformers-4.8.2-py3-none-any.whl (2.5MB)\n",
            "\u001b[K     |████████████████████████████████| 2.5MB 7.9MB/s \n",
            "\u001b[?25hCollecting accelerate\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f7/fa/d173d923c953d930702066894abf128a7e5258c6f64cf088d2c5a83f46a3/accelerate-0.3.0-py3-none-any.whl (49kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 7.6MB/s \n",
            "\u001b[?25hCollecting datasets\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/08/a2/d4e1024c891506e1cee8f9d719d20831bac31cb5b7416983c4d2f65a6287/datasets-1.8.0-py3-none-any.whl (237kB)\n",
            "\u001b[K     |████████████████████████████████| 245kB 60.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.41.1)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/ee/67241dc87f266093c533a2d4d3d69438e57d7a90abb216fa076e7d475d4a/sacremoses-0.0.45-py3-none-any.whl (895kB)\n",
            "\u001b[K     |████████████████████████████████| 901kB 42.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from transformers) (3.13)\n",
            "Collecting tokenizers<0.11,>=0.10.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/e2/df3543e8ffdab68f5acc73f613de9c2b155ac47f162e725dcac87c521c11/tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3MB)\n",
            "\u001b[K     |████████████████████████████████| 3.3MB 43.2MB/s \n",
            "\u001b[?25hCollecting huggingface-hub==0.0.12\n",
            "  Downloading https://files.pythonhosted.org/packages/2f/ee/97e253668fda9b17e968b3f97b2f8e53aa0127e8807d24a547687423fe0b/huggingface_hub-0.0.12-py3-none-any.whl\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers) (4.5.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers) (20.9)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Collecting pyaml>=20.4.0\n",
            "  Downloading https://files.pythonhosted.org/packages/15/c4/1310a054d33abc318426a956e7d6df0df76a6ddfa9c66f6310274fb75d42/pyaml-20.4.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from accelerate) (1.9.0+cu102)\n",
            "Requirement already satisfied: pyarrow<4.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from datasets) (3.0.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets) (0.70.12.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets) (1.1.5)\n",
            "Collecting fsspec\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0e/3a/666e63625a19883ae8e1674099e631f9737bd5478c4790e5ad49c5ac5261/fsspec-2021.6.1-py3-none-any.whl (115kB)\n",
            "\u001b[K     |████████████████████████████████| 122kB 60.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets) (0.3.4)\n",
            "Collecting xxhash\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/4f/0a862cad26aa2ed7a7cd87178cbbfa824fc1383e472d63596a0d018374e7/xxhash-2.0.2-cp37-cp37m-manylinux2010_x86_64.whl (243kB)\n",
            "\u001b[K     |████████████████████████████████| 245kB 54.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.5.30)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.0.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from huggingface-hub==0.0.12->transformers) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.4.1)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2018.9)\n",
            "Installing collected packages: sacremoses, tokenizers, huggingface-hub, transformers, pyaml, accelerate, fsspec, xxhash, datasets\n",
            "Successfully installed accelerate-0.3.0 datasets-1.8.0 fsspec-2021.6.1 huggingface-hub-0.0.12 pyaml-20.4.0 sacremoses-0.0.45 tokenizers-0.10.3 transformers-4.8.2 xxhash-2.0.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wZYxy_hAXV5e"
      },
      "source": [
        "import os\n",
        "import math\n",
        "import random\n",
        "import time\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.utils.data import Dataset\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "from transformers import AdamW\n",
        "from transformers import AutoTokenizer\n",
        "from transformers import AutoModel\n",
        "from transformers import AutoConfig\n",
        "from transformers import get_cosine_schedule_with_warmup\n",
        "\n",
        "from sklearn.model_selection import KFold\n",
        "\n",
        "import gc\n",
        "gc.enable()"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6-8wniOeBzQ"
      },
      "source": [
        "# Set constant"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DQjr2-NPXV5f"
      },
      "source": [
        "NUM_FOLDS = 5\n",
        "NUM_EPOCHS = 3\n",
        "BATCH_SIZE = 16\n",
        "MAX_LEN = 248\n",
        "#(eval_rmse, step_size)\n",
        "EVAL_SCHEDULE = [(0.50, 16), (0.49, 8), (0.48, 4), (0.47, 2), (-1., 1)]\n",
        "ROBERTA_PATH = \"../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/\"\n",
        "TOKENIZER_PATH = \"../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/\"\n",
        "#ROBERTA_PATH = \"../input/clrp-roberta-base/clrp_roberta_base\"\n",
        "#TOKENIZER_PATH = \"../input/clrp-roberta-base/clrp_roberta_base\"\n",
        "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\""
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tBgC9X-JOotk",
        "outputId": "a7df7fc9-01d2-4bca-c881-6f0f3f0d767c"
      },
      "source": [
        "EVAL_SCHEDULE[0][1]"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "16"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xnzprhriHc7B"
      },
      "source": [
        "# Define utility functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZNCjd5KEXV5f"
      },
      "source": [
        "def set_random_seed(random_seed):\n",
        "    random.seed(random_seed)\n",
        "    np.random.seed(random_seed)\n",
        "    os.environ[\"PYTHONHASHSEED\"] = str(random_seed)\n",
        "\n",
        "    torch.manual_seed(random_seed)\n",
        "    torch.cuda.manual_seed(random_seed)\n",
        "    torch.cuda.manual_seed_all(random_seed)\n",
        "\n",
        "    torch.backends.cudnn.deterministic = True"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A95IJgC8WPLz"
      },
      "source": [
        "train_dfには、Stratified kfold済みのデータセットを利用する。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T-5y27aJXV5f"
      },
      "source": [
        "#Use stratified k-fold train dataset\n",
        "#train_df = pd.read_csv(\"/kaggle/input/commonlitreadabilityprize/train.csv\")\n",
        "train_df = pd.read_csv(\"../input/commonlitreadabilityprize/train_stratiKfold.csv\")\n",
        "\n",
        "# Remove incomplete entries if any.\n",
        "train_df.drop(train_df[(train_df.target == 0) & (train_df.standard_error == 0)].index,\n",
        "              inplace=True)\n",
        "train_df.reset_index(drop=True, inplace=True)\n",
        "\n",
        "test_df = pd.read_csv(\"../input/commonlitreadabilityprize/test.csv\")\n",
        "submission_df = pd.read_csv(\"../input/commonlitreadabilityprize/sample_submission.csv\")"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bR2QTnuIXV5g"
      },
      "source": [
        "#TokenizerはRoberta-baseと同じ\n",
        "tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH)"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "uq6O3LeXZ3Q9",
        "outputId": "d72154c2-8d14-4543-fce0-784a7b99e823"
      },
      "source": [
        "train_df[train_df['kfold']!=1]"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>id</th>\n",
              "      <th>url_legal</th>\n",
              "      <th>license</th>\n",
              "      <th>excerpt</th>\n",
              "      <th>target</th>\n",
              "      <th>standard_error</th>\n",
              "      <th>kfold</th>\n",
              "      <th>bins</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>bf24448fb</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Anywhere there is a frontier, where there are ...</td>\n",
              "      <td>-1.866238</td>\n",
              "      <td>0.510911</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>7cad0f936</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A great violinist, Ole Bull by name, visited t...</td>\n",
              "      <td>-0.578482</td>\n",
              "      <td>0.471768</td>\n",
              "      <td>2</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>91e87e7dc</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Hans stopped snoring and awoke at supper-time....</td>\n",
              "      <td>-0.186015</td>\n",
              "      <td>0.492731</td>\n",
              "      <td>2</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5</td>\n",
              "      <td>20a9f9032</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>The Government of the United States has viewed...</td>\n",
              "      <td>-1.391438</td>\n",
              "      <td>0.499195</td>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>6</td>\n",
              "      <td>daab29b47</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forty years ago women were given no representa...</td>\n",
              "      <td>-1.291128</td>\n",
              "      <td>0.531642</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2826</th>\n",
              "      <td>2827</td>\n",
              "      <td>d25b7c3aa</td>\n",
              "      <td>https://www.commonlit.org/texts/the-center-of-...</td>\n",
              "      <td>CC BY-NC-SA 2.0</td>\n",
              "      <td>The sun is a star, just like the other million...</td>\n",
              "      <td>-0.580631</td>\n",
              "      <td>0.457745</td>\n",
              "      <td>2</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2828</th>\n",
              "      <td>2829</td>\n",
              "      <td>3c1662f6d</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>It was the northwest coast of Australia, the c...</td>\n",
              "      <td>-1.678689</td>\n",
              "      <td>0.493150</td>\n",
              "      <td>2</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2830</th>\n",
              "      <td>2831</td>\n",
              "      <td>64b635d77</td>\n",
              "      <td>https://www.africanstorybook.org/</td>\n",
              "      <td>CC BY 4.0</td>\n",
              "      <td>Once long ago, the birds had a meeting. They w...</td>\n",
              "      <td>0.639650</td>\n",
              "      <td>0.503652</td>\n",
              "      <td>2</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2831</th>\n",
              "      <td>2832</td>\n",
              "      <td>d6764322c</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>As an adult, I might learn new actions by taki...</td>\n",
              "      <td>1.024258</td>\n",
              "      <td>0.549119</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2832</th>\n",
              "      <td>2833</td>\n",
              "      <td>cf44e1a67</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Where do we feel empathy and how does it work?...</td>\n",
              "      <td>0.564179</td>\n",
              "      <td>0.486699</td>\n",
              "      <td>2</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2266 rows × 9 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      Unnamed: 0         id  ... kfold bins\n",
              "1              1  bf24448fb  ...     3    4\n",
              "2              2  7cad0f936  ...     2    6\n",
              "4              4  91e87e7dc  ...     2    7\n",
              "5              5  20a9f9032  ...     4    5\n",
              "6              6  daab29b47  ...     2    5\n",
              "...          ...        ...  ...   ...  ...\n",
              "2826        2827  d25b7c3aa  ...     2    6\n",
              "2828        2829  3c1662f6d  ...     2    4\n",
              "2830        2831  64b635d77  ...     2    9\n",
              "2831        2832  d6764322c  ...     3   10\n",
              "2832        2833  cf44e1a67  ...     2    9\n",
              "\n",
              "[2266 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b5miW_QBacqW"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "luovjtC0XV5g"
      },
      "source": [
        "# Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LHVj3D05XV5g"
      },
      "source": [
        "class LitDataset(Dataset):\n",
        "    def __init__(self, df, inference_only=False):\n",
        "        super().__init__()\n",
        "\n",
        "        self.df = df        \n",
        "        self.inference_only = inference_only\n",
        "        self.text = df.excerpt.tolist()\n",
        "        #改行を消してみる。元のNotebookではここはコメントアウトされている\n",
        "        #self.text = [text.replace(\"\\n\", \" \") for text in self.text]\n",
        "        \n",
        "        if not self.inference_only:\n",
        "            self.target = torch.tensor(df.target.values, dtype=torch.float32)        \n",
        "    \n",
        "        self.encoded = tokenizer.batch_encode_plus(\n",
        "            self.text,\n",
        "            padding = 'max_length',            \n",
        "            max_length = MAX_LEN,\n",
        "            truncation = True,\n",
        "            return_attention_mask=True\n",
        "        )        \n",
        " \n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.df)\n",
        "\n",
        "    \n",
        "    def __getitem__(self, index):        \n",
        "        input_ids = torch.tensor(self.encoded['input_ids'][index])\n",
        "        attention_mask = torch.tensor(self.encoded['attention_mask'][index])\n",
        "        \n",
        "        if self.inference_only:\n",
        "            return (input_ids, attention_mask)            \n",
        "        else:\n",
        "            target = self.target[index]\n",
        "            return (input_ids, attention_mask, target)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Smttt02WXV5h"
      },
      "source": [
        "# Model\n",
        "The model is inspired by the one from [Maunish](https://www.kaggle.com/maunish/clrp-roberta-svm)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RL6qw55sXV5h"
      },
      "source": [
        "class LitModel(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "\n",
        "        config = AutoConfig.from_pretrained(ROBERTA_PATH)\n",
        "        #config.jsonに書いてある設定値を更新する\n",
        "        config.update({\"output_hidden_states\":True, \n",
        "                       \"hidden_dropout_prob\": 0.0,\n",
        "                       \"layer_norm_eps\": 1e-7})                       \n",
        "        \n",
        "        self.roberta = AutoModel.from_pretrained(ROBERTA_PATH, config=config)  \n",
        "            \n",
        "        self.attention = nn.Sequential(            \n",
        "            nn.Linear(768, 512),            \n",
        "            nn.Tanh(),                       \n",
        "            nn.Linear(512, 1),\n",
        "            nn.Softmax(dim=1)\n",
        "        )        \n",
        "\n",
        "        self.regressor = nn.Sequential(                        \n",
        "            nn.Linear(768, 1)                        \n",
        "        )\n",
        "        \n",
        "\n",
        "    def forward(self, input_ids, attention_mask):\n",
        "        roberta_output = self.roberta(input_ids=input_ids,\n",
        "                                      attention_mask=attention_mask)        \n",
        "\n",
        "        # There are a total of 13 layers of hidden states.\n",
        "        # 1 for the embedding layer, and 12 for the 12 Roberta layers.\n",
        "        # We take the hidden states from the last Roberta layer.\n",
        "        last_layer_hidden_states = roberta_output.hidden_states[-1]\n",
        "\n",
        "        # The number of cells is MAX_LEN.\n",
        "        # The size of the hidden state of each cell is 768 (for roberta-base).\n",
        "        # In order to condense hidden states of all cells to a context vector,\n",
        "        # we compute a weighted average of the hidden states of all cells.\n",
        "        # We compute the weight of each cell, using the attention neural network.\n",
        "        weights = self.attention(last_layer_hidden_states)\n",
        "                \n",
        "        # weights.shape is BATCH_SIZE x MAX_LEN x 1\n",
        "        # last_layer_hidden_states.shape is BATCH_SIZE x MAX_LEN x 768        \n",
        "        # Now we compute context_vector as the weighted average.\n",
        "        # context_vector.shape is BATCH_SIZE x 768\n",
        "        context_vector = torch.sum(weights * last_layer_hidden_states, dim=1)        \n",
        "        \n",
        "        # Now we reduce the context vector to the prediction score.\n",
        "        return self.regressor(context_vector)"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t_BKdywXVDzk"
      },
      "source": [
        "## Define eval"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FrZm0bLLXV5h"
      },
      "source": [
        "#MSEで評価\n",
        "def eval_mse(model, data_loader):\n",
        "    \"\"\"Evaluates the mean squared error of the |model| on |data_loader|\"\"\"\n",
        "    model.eval()            \n",
        "    mse_sum = 0\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for batch_num, (input_ids, attention_mask, target) in enumerate(data_loader):\n",
        "            input_ids = input_ids.to(DEVICE)\n",
        "            attention_mask = attention_mask.to(DEVICE)                        \n",
        "            target = target.to(DEVICE)           \n",
        "            \n",
        "            pred = model(input_ids, attention_mask)                       \n",
        "\n",
        "            mse_sum += nn.MSELoss(reduction=\"sum\")(pred.flatten(), target).item()\n",
        "                \n",
        "\n",
        "    return mse_sum / len(data_loader.dataset)"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yJsYZflrVBMr"
      },
      "source": [
        "## Define predict"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JxqWSjMHXV5i"
      },
      "source": [
        "def predict(model, data_loader):\n",
        "    \"\"\"Returns an np.array with predictions of the |model| on |data_loader|\"\"\"\n",
        "    model.eval()\n",
        "\n",
        "    result = np.zeros(len(data_loader.dataset))    \n",
        "    index = 0\n",
        "    \n",
        "    with torch.no_grad():\n",
        "        for batch_num, (input_ids, attention_mask) in enumerate(data_loader):\n",
        "            input_ids = input_ids.to(DEVICE)\n",
        "            attention_mask = attention_mask.to(DEVICE)\n",
        "                        \n",
        "            pred = model(input_ids, attention_mask)                        \n",
        "\n",
        "            result[index : index + pred.shape[0]] = pred.flatten().to(\"cpu\")\n",
        "            index += pred.shape[0]\n",
        "\n",
        "    return result"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vdOQdEY7U-gl"
      },
      "source": [
        "### Define Train"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G7fx532GXV5i"
      },
      "source": [
        "def train(model, model_path, train_loader, val_loader,\n",
        "          optimizer, scheduler=None, num_epochs=NUM_EPOCHS):    \n",
        "    best_val_rmse = None\n",
        "    best_epoch = 0\n",
        "    step = 0\n",
        "    last_eval_step = 0\n",
        "    #EVAL_SCHEDULE = [(0.50, 16), (0.49, 8), (0.48, 4), (0.47, 2), (-1., 1)]\n",
        "    #-> EVAL_SCHEDULE[0][1] = 16\n",
        "    eval_period = EVAL_SCHEDULE[0][1]    \n",
        "\n",
        "    start = time.time()\n",
        "\n",
        "    #Epoch数だけ繰り返す\n",
        "    for epoch in range(num_epochs):                           \n",
        "        val_rmse = None         \n",
        "\n",
        "        for batch_num, (input_ids, attention_mask, target) in enumerate(train_loader):\n",
        "            input_ids = input_ids.to(DEVICE)\n",
        "            attention_mask = attention_mask.to(DEVICE)            \n",
        "            target = target.to(DEVICE)                        \n",
        "\n",
        "            optimizer.zero_grad()\n",
        "            \n",
        "            model.train()\n",
        "\n",
        "            pred = model(input_ids, attention_mask)\n",
        "                                                        \n",
        "            mse = nn.MSELoss(reduction=\"mean\")(pred.flatten(), target)\n",
        "                        \n",
        "            mse.backward()\n",
        "\n",
        "            #https://stackoverflow.com/questions/60120043/optimizer-and-scheduler-for-bert-fine-tuning\n",
        "            #`optimizer.step()`の直後、`scheduler.step()`をすべてのバッチで呼び出して、学習率を更新します。\n",
        "            optimizer.step()\n",
        "            if scheduler:\n",
        "                scheduler.step()\n",
        "            \n",
        "            #eval_period(初期値は16）stepごとにRMSEを評価\n",
        "            if step >= last_eval_step + eval_period:\n",
        "                # Evaluate the model on val_loader.\n",
        "                elapsed_seconds = time.time() - start\n",
        "                num_steps = step - last_eval_step\n",
        "                print(f\"\\n{num_steps} steps took {elapsed_seconds:0.3} seconds\")\n",
        "                last_eval_step = step\n",
        "                \n",
        "                val_rmse = math.sqrt(eval_mse(model, val_loader))                            \n",
        "\n",
        "                print(f\"Epoch: {epoch} batch_num: {batch_num}\", \n",
        "                      f\"val_rmse: {val_rmse:0.4}\")\n",
        "\n",
        "                #EVAL_SCHEDULEに定義したrmseによって\n",
        "                #eval_periodを変更する\n",
        "                for rmse, period in EVAL_SCHEDULE:\n",
        "                    if val_rmse >= rmse:\n",
        "                        eval_period = period\n",
        "                        break                               \n",
        "                \n",
        "                #ベストスコアを記録\n",
        "                if not best_val_rmse or val_rmse < best_val_rmse:                    \n",
        "                    best_val_rmse = val_rmse\n",
        "                    best_epoch = epoch\n",
        "                    torch.save(model.state_dict(), model_path)\n",
        "                    print(f\"New best_val_rmse: {best_val_rmse:0.4}\")\n",
        "                else:       \n",
        "                    print(f\"Still best_val_rmse: {best_val_rmse:0.4}\",\n",
        "                          f\"(from epoch {best_epoch})\")                                    \n",
        "                    \n",
        "                start = time.time()\n",
        "\n",
        "            #stepをインクリメント                                          \n",
        "            step += 1\n",
        "                        \n",
        "    \n",
        "    return best_val_rmse"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U1bmfWQiU4yT"
      },
      "source": [
        "## Create Optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q8nGcWc7XV5j"
      },
      "source": [
        "def create_optimizer(model):\n",
        "    named_parameters = list(model.named_parameters())    \n",
        "    \n",
        "    roberta_parameters = named_parameters[:197]    \n",
        "    attention_parameters = named_parameters[199:203]\n",
        "    regressor_parameters = named_parameters[203:]\n",
        "        \n",
        "    attention_group = [params for (name, params) in attention_parameters]\n",
        "    regressor_group = [params for (name, params) in regressor_parameters]\n",
        "\n",
        "    parameters = []\n",
        "    parameters.append({\"params\": attention_group})\n",
        "    parameters.append({\"params\": regressor_group})\n",
        "\n",
        "    for layer_num, (name, params) in enumerate(roberta_parameters):\n",
        "        weight_decay = 0.0 if \"bias\" in name else 0.01\n",
        "\n",
        "        lr = 2e-5\n",
        "\n",
        "        if layer_num >= 69:        \n",
        "            lr = 5e-5\n",
        "\n",
        "        if layer_num >= 133:\n",
        "            lr = 1e-4\n",
        "\n",
        "        parameters.append({\"params\": params,\n",
        "                           \"weight_decay\": weight_decay,\n",
        "                           \"lr\": lr})\n",
        "\n",
        "    return AdamW(parameters)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QW8_ta8Shvbp"
      },
      "source": [
        "## Run"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0ASAFbdqXV5k",
        "outputId": "36b3ac03-6017-4af2-980b-86797671a4d0"
      },
      "source": [
        "gc.collect()\n",
        "\n",
        "SEED = 1000\n",
        "list_val_rmse = []\n",
        "\n",
        "for fold in range(NUM_FOLDS): \n",
        "    print(f\"\\nFold {fold + 1}/{NUM_FOLDS}\")\n",
        "    model_path = f\"model_{fold + 1}.pth\"\n",
        "        \n",
        "    set_random_seed(SEED + fold)\n",
        "\n",
        "    #Stratified kfold train dataset用に修正\n",
        "    train_dataset = LitDataset(train_df[train_df['kfold']!=fold])    \n",
        "    val_dataset = LitDataset(train_df[train_df['kfold']==fold])    \n",
        "    \n",
        "    #https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader\n",
        "    train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE,\n",
        "                              drop_last=True, shuffle=True, num_workers=2)    \n",
        "    val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE,\n",
        "                            drop_last=False, shuffle=False, num_workers=2)    \n",
        "    \n",
        "    #random_seedは、Foldごとに変わる\n",
        "    set_random_seed(SEED + fold)    \n",
        "    \n",
        "    model = LitModel().to(DEVICE)\n",
        "    \n",
        "    optimizer = create_optimizer(model)\n",
        "    #Schedulerには、get_cosine_schedule_with_warmupを使っている\n",
        "    #その他の選択肢: https://huggingface.co/transformers/main_classes/optimizer_schedules.html#schedules                        \n",
        "    scheduler = get_cosine_schedule_with_warmup(\n",
        "        optimizer,\n",
        "        num_training_steps=NUM_EPOCHS * len(train_loader),\n",
        "        num_warmup_steps=50)    \n",
        "    \n",
        "    list_val_rmse.append(train(model, model_path, train_loader,\n",
        "                               val_loader, optimizer, scheduler=scheduler))\n",
        "\n",
        "    del model\n",
        "    gc.collect()\n",
        "    \n",
        "    print(\"\\nPerformance estimates:\")\n",
        "    print(list_val_rmse)\n",
        "    print(\"Mean:\", np.array(list_val_rmse).mean())\n",
        "\n",
        "    "
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Fold 1/5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "16 steps took 7.07 seconds\n",
            "Epoch: 0 batch_num: 16 val_rmse: 0.9574\n",
            "New best_val_rmse: 0.9574\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 32 val_rmse: 0.7136\n",
            "New best_val_rmse: 0.7136\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 48 val_rmse: 0.6355\n",
            "New best_val_rmse: 0.6355\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 64 val_rmse: 0.6181\n",
            "New best_val_rmse: 0.6181\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 80 val_rmse: 0.6541\n",
            "Still best_val_rmse: 0.6181 (from epoch 0)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 96 val_rmse: 0.5713\n",
            "New best_val_rmse: 0.5713\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 112 val_rmse: 0.5811\n",
            "Still best_val_rmse: 0.5713 (from epoch 0)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 0 batch_num: 128 val_rmse: 0.5241\n",
            "New best_val_rmse: 0.5241\n",
            "\n",
            "16 steps took 6.46 seconds\n",
            "Epoch: 1 batch_num: 3 val_rmse: 0.5121\n",
            "New best_val_rmse: 0.5121\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 19 val_rmse: 0.514\n",
            "Still best_val_rmse: 0.5121 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 35 val_rmse: 0.5455\n",
            "Still best_val_rmse: 0.5121 (from epoch 1)\n",
            "\n",
            "16 steps took 6.34 seconds\n",
            "Epoch: 1 batch_num: 51 val_rmse: 0.4907\n",
            "New best_val_rmse: 0.4907\n",
            "\n",
            "8 steps took 3.17 seconds\n",
            "Epoch: 1 batch_num: 59 val_rmse: 0.5458\n",
            "Still best_val_rmse: 0.4907 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 75 val_rmse: 0.4984\n",
            "Still best_val_rmse: 0.4907 (from epoch 1)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 1 batch_num: 83 val_rmse: 0.5508\n",
            "Still best_val_rmse: 0.4907 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 99 val_rmse: 0.4975\n",
            "Still best_val_rmse: 0.4907 (from epoch 1)\n",
            "\n",
            "8 steps took 3.17 seconds\n",
            "Epoch: 1 batch_num: 107 val_rmse: 0.505\n",
            "Still best_val_rmse: 0.4907 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 123 val_rmse: 0.549\n",
            "Still best_val_rmse: 0.4907 (from epoch 1)\n",
            "\n",
            "16 steps took 6.34 seconds\n",
            "Epoch: 1 batch_num: 139 val_rmse: 0.4813\n",
            "New best_val_rmse: 0.4813\n",
            "\n",
            "4 steps took 1.74 seconds\n",
            "Epoch: 2 batch_num: 2 val_rmse: 0.4882\n",
            "Still best_val_rmse: 0.4813 (from epoch 1)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 6 val_rmse: 0.4962\n",
            "Still best_val_rmse: 0.4813 (from epoch 1)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 2 batch_num: 14 val_rmse: 0.4795\n",
            "New best_val_rmse: 0.4795\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 2 batch_num: 16 val_rmse: 0.4827\n",
            "Still best_val_rmse: 0.4795 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 20 val_rmse: 0.4816\n",
            "Still best_val_rmse: 0.4795 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 24 val_rmse: 0.4777\n",
            "New best_val_rmse: 0.4777\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 26 val_rmse: 0.4755\n",
            "New best_val_rmse: 0.4755\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 2 batch_num: 28 val_rmse: 0.4808\n",
            "Still best_val_rmse: 0.4755 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 32 val_rmse: 0.4784\n",
            "Still best_val_rmse: 0.4755 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 34 val_rmse: 0.4751\n",
            "New best_val_rmse: 0.4751\n",
            "\n",
            "2 steps took 0.798 seconds\n",
            "Epoch: 2 batch_num: 36 val_rmse: 0.4739\n",
            "New best_val_rmse: 0.4739\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 38 val_rmse: 0.4729\n",
            "New best_val_rmse: 0.4729\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 40 val_rmse: 0.4725\n",
            "New best_val_rmse: 0.4725\n",
            "\n",
            "2 steps took 0.797 seconds\n",
            "Epoch: 2 batch_num: 42 val_rmse: 0.472\n",
            "New best_val_rmse: 0.472\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 44 val_rmse: 0.4715\n",
            "New best_val_rmse: 0.4715\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 46 val_rmse: 0.4718\n",
            "Still best_val_rmse: 0.4715 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 48 val_rmse: 0.4752\n",
            "Still best_val_rmse: 0.4715 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 50 val_rmse: 0.4781\n",
            "Still best_val_rmse: 0.4715 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 52 val_rmse: 0.4769\n",
            "Still best_val_rmse: 0.4715 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 54 val_rmse: 0.4734\n",
            "Still best_val_rmse: 0.4715 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 56 val_rmse: 0.4719\n",
            "Still best_val_rmse: 0.4715 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 58 val_rmse: 0.4714\n",
            "New best_val_rmse: 0.4714\n",
            "\n",
            "2 steps took 0.795 seconds\n",
            "Epoch: 2 batch_num: 60 val_rmse: 0.4716\n",
            "Still best_val_rmse: 0.4714 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 62 val_rmse: 0.4712\n",
            "New best_val_rmse: 0.4712\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 2 batch_num: 64 val_rmse: 0.4711\n",
            "New best_val_rmse: 0.4711\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 2 batch_num: 66 val_rmse: 0.4717\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 2 batch_num: 68 val_rmse: 0.4732\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 70 val_rmse: 0.474\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 2 batch_num: 72 val_rmse: 0.4737\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 74 val_rmse: 0.4742\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 76 val_rmse: 0.4736\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 78 val_rmse: 0.4738\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 80 val_rmse: 0.4738\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 82 val_rmse: 0.4738\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 84 val_rmse: 0.474\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 86 val_rmse: 0.475\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 88 val_rmse: 0.4762\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 90 val_rmse: 0.4764\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 92 val_rmse: 0.4764\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 94 val_rmse: 0.4754\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 96 val_rmse: 0.4743\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 98 val_rmse: 0.4739\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 100 val_rmse: 0.4737\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 102 val_rmse: 0.4734\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 104 val_rmse: 0.4736\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 106 val_rmse: 0.4744\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 108 val_rmse: 0.4752\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 110 val_rmse: 0.4756\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 112 val_rmse: 0.4757\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 114 val_rmse: 0.4759\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 116 val_rmse: 0.4756\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 118 val_rmse: 0.4754\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 120 val_rmse: 0.4752\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 2 batch_num: 122 val_rmse: 0.4751\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 124 val_rmse: 0.4751\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 126 val_rmse: 0.4751\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 128 val_rmse: 0.4752\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.796 seconds\n",
            "Epoch: 2 batch_num: 130 val_rmse: 0.4752\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 132 val_rmse: 0.4752\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 134 val_rmse: 0.4752\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 2 batch_num: 136 val_rmse: 0.4751\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.785 seconds\n",
            "Epoch: 2 batch_num: 138 val_rmse: 0.4751\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 140 val_rmse: 0.4751\n",
            "Still best_val_rmse: 0.4711 (from epoch 2)\n",
            "\n",
            "Performance estimates:\n",
            "[0.471089345660429]\n",
            "Mean: 0.471089345660429\n",
            "\n",
            "Fold 2/5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "16 steps took 6.84 seconds\n",
            "Epoch: 0 batch_num: 16 val_rmse: 1.027\n",
            "New best_val_rmse: 1.027\n",
            "\n",
            "16 steps took 6.34 seconds\n",
            "Epoch: 0 batch_num: 32 val_rmse: 0.7094\n",
            "New best_val_rmse: 0.7094\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 48 val_rmse: 0.8308\n",
            "Still best_val_rmse: 0.7094 (from epoch 0)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 64 val_rmse: 0.6959\n",
            "New best_val_rmse: 0.6959\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 80 val_rmse: 0.6494\n",
            "New best_val_rmse: 0.6494\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 96 val_rmse: 0.5598\n",
            "New best_val_rmse: 0.5598\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 112 val_rmse: 0.5814\n",
            "Still best_val_rmse: 0.5598 (from epoch 0)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 128 val_rmse: 0.5336\n",
            "New best_val_rmse: 0.5336\n",
            "\n",
            "16 steps took 6.47 seconds\n",
            "Epoch: 1 batch_num: 3 val_rmse: 0.5179\n",
            "New best_val_rmse: 0.5179\n",
            "\n",
            "16 steps took 6.34 seconds\n",
            "Epoch: 1 batch_num: 19 val_rmse: 0.5396\n",
            "Still best_val_rmse: 0.5179 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 35 val_rmse: 0.505\n",
            "New best_val_rmse: 0.505\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 1 batch_num: 51 val_rmse: 0.5113\n",
            "Still best_val_rmse: 0.505 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 67 val_rmse: 0.5496\n",
            "Still best_val_rmse: 0.505 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 83 val_rmse: 0.61\n",
            "Still best_val_rmse: 0.505 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 99 val_rmse: 0.4962\n",
            "New best_val_rmse: 0.4962\n",
            "\n",
            "8 steps took 3.17 seconds\n",
            "Epoch: 1 batch_num: 107 val_rmse: 0.4942\n",
            "New best_val_rmse: 0.4942\n",
            "\n",
            "8 steps took 3.17 seconds\n",
            "Epoch: 1 batch_num: 115 val_rmse: 0.5149\n",
            "Still best_val_rmse: 0.4942 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 131 val_rmse: 0.5029\n",
            "Still best_val_rmse: 0.4942 (from epoch 1)\n",
            "\n",
            "16 steps took 6.48 seconds\n",
            "Epoch: 2 batch_num: 6 val_rmse: 0.4899\n",
            "New best_val_rmse: 0.4899\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 10 val_rmse: 0.4826\n",
            "New best_val_rmse: 0.4826\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 14 val_rmse: 0.4959\n",
            "Still best_val_rmse: 0.4826 (from epoch 2)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 2 batch_num: 22 val_rmse: 0.5019\n",
            "Still best_val_rmse: 0.4826 (from epoch 2)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 38 val_rmse: 0.4801\n",
            "New best_val_rmse: 0.4801\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 42 val_rmse: 0.4792\n",
            "New best_val_rmse: 0.4792\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 2 batch_num: 44 val_rmse: 0.4812\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 48 val_rmse: 0.4811\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 52 val_rmse: 0.4797\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 54 val_rmse: 0.4796\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 56 val_rmse: 0.4807\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 60 val_rmse: 0.4796\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 62 val_rmse: 0.4796\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 64 val_rmse: 0.4799\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "2 steps took 0.792 seconds\n",
            "Epoch: 2 batch_num: 66 val_rmse: 0.4801\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 70 val_rmse: 0.4807\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 74 val_rmse: 0.4811\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 78 val_rmse: 0.4818\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 82 val_rmse: 0.4835\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 86 val_rmse: 0.4816\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 90 val_rmse: 0.4826\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 94 val_rmse: 0.4869\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 98 val_rmse: 0.4887\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.59 seconds\n",
            "Epoch: 2 batch_num: 102 val_rmse: 0.4857\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 106 val_rmse: 0.4832\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 110 val_rmse: 0.4816\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 114 val_rmse: 0.481\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 118 val_rmse: 0.4809\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 122 val_rmse: 0.4808\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.59 seconds\n",
            "Epoch: 2 batch_num: 126 val_rmse: 0.4808\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 130 val_rmse: 0.4808\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 134 val_rmse: 0.4808\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 138 val_rmse: 0.4808\n",
            "Still best_val_rmse: 0.4792 (from epoch 2)\n",
            "\n",
            "Performance estimates:\n",
            "[0.471089345660429, 0.4791831833542291]\n",
            "Mean: 0.47513626450732904\n",
            "\n",
            "Fold 3/5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "16 steps took 6.84 seconds\n",
            "Epoch: 0 batch_num: 16 val_rmse: 0.9434\n",
            "New best_val_rmse: 0.9434\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 32 val_rmse: 0.8244\n",
            "New best_val_rmse: 0.8244\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 48 val_rmse: 0.6613\n",
            "New best_val_rmse: 0.6613\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 0 batch_num: 64 val_rmse: 0.5816\n",
            "New best_val_rmse: 0.5816\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 80 val_rmse: 0.6065\n",
            "Still best_val_rmse: 0.5816 (from epoch 0)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 96 val_rmse: 0.5704\n",
            "New best_val_rmse: 0.5704\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 112 val_rmse: 0.526\n",
            "New best_val_rmse: 0.526\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 128 val_rmse: 0.5661\n",
            "Still best_val_rmse: 0.526 (from epoch 0)\n",
            "\n",
            "16 steps took 6.46 seconds\n",
            "Epoch: 1 batch_num: 3 val_rmse: 0.4947\n",
            "New best_val_rmse: 0.4947\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 1 batch_num: 11 val_rmse: 0.5053\n",
            "Still best_val_rmse: 0.4947 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 27 val_rmse: 0.484\n",
            "New best_val_rmse: 0.484\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 1 batch_num: 31 val_rmse: 0.507\n",
            "Still best_val_rmse: 0.484 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 47 val_rmse: 0.4669\n",
            "New best_val_rmse: 0.4669\n",
            "\n",
            "1 steps took 0.395 seconds\n",
            "Epoch: 1 batch_num: 48 val_rmse: 0.4812\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 1 batch_num: 52 val_rmse: 0.5085\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 1 batch_num: 68 val_rmse: 0.5238\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 1 batch_num: 84 val_rmse: 0.4703\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 1 batch_num: 86 val_rmse: 0.478\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 1 batch_num: 88 val_rmse: 0.4754\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 1 batch_num: 90 val_rmse: 0.4712\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 1 batch_num: 92 val_rmse: 0.516\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 1 batch_num: 108 val_rmse: 0.4758\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 1 batch_num: 110 val_rmse: 0.4739\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 1 batch_num: 112 val_rmse: 0.5057\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 128 val_rmse: 0.4711\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 1 batch_num: 130 val_rmse: 0.4836\n",
            "Still best_val_rmse: 0.4669 (from epoch 1)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 1 batch_num: 134 val_rmse: 0.4596\n",
            "New best_val_rmse: 0.4596\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 1 batch_num: 135 val_rmse: 0.4592\n",
            "New best_val_rmse: 0.4592\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 1 batch_num: 136 val_rmse: 0.4647\n",
            "Still best_val_rmse: 0.4592 (from epoch 1)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 1 batch_num: 137 val_rmse: 0.4702\n",
            "Still best_val_rmse: 0.4592 (from epoch 1)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 1 batch_num: 139 val_rmse: 0.4581\n",
            "New best_val_rmse: 0.4581\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 1 batch_num: 140 val_rmse: 0.4591\n",
            "Still best_val_rmse: 0.4581 (from epoch 1)\n",
            "\n",
            "1 steps took 0.555 seconds\n",
            "Epoch: 2 batch_num: 0 val_rmse: 0.4664\n",
            "Still best_val_rmse: 0.4581 (from epoch 1)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 1 val_rmse: 0.4756\n",
            "Still best_val_rmse: 0.4581 (from epoch 1)\n",
            "\n",
            "2 steps took 0.785 seconds\n",
            "Epoch: 2 batch_num: 3 val_rmse: 0.4855\n",
            "Still best_val_rmse: 0.4581 (from epoch 1)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 7 val_rmse: 0.458\n",
            "New best_val_rmse: 0.458\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 8 val_rmse: 0.4599\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 9 val_rmse: 0.464\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 10 val_rmse: 0.4633\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 11 val_rmse: 0.4618\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 12 val_rmse: 0.4607\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 13 val_rmse: 0.4599\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 14 val_rmse: 0.4599\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 15 val_rmse: 0.4614\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 16 val_rmse: 0.4636\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 17 val_rmse: 0.4683\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 18 val_rmse: 0.4718\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 20 val_rmse: 0.4759\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 22 val_rmse: 0.4702\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 24 val_rmse: 0.4636\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 25 val_rmse: 0.4635\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.396 seconds\n",
            "Epoch: 2 batch_num: 26 val_rmse: 0.4635\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 27 val_rmse: 0.4645\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 28 val_rmse: 0.4643\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 29 val_rmse: 0.4627\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 30 val_rmse: 0.4611\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 31 val_rmse: 0.4607\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 32 val_rmse: 0.4614\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 33 val_rmse: 0.4649\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 34 val_rmse: 0.4727\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 36 val_rmse: 0.4818\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 40 val_rmse: 0.4753\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 42 val_rmse: 0.4716\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 44 val_rmse: 0.4625\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 45 val_rmse: 0.4592\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 46 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.458 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 47 val_rmse: 0.4575\n",
            "New best_val_rmse: 0.4575\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 48 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 49 val_rmse: 0.4594\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 50 val_rmse: 0.4627\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 51 val_rmse: 0.4663\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 52 val_rmse: 0.4692\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 53 val_rmse: 0.4717\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 55 val_rmse: 0.4756\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 57 val_rmse: 0.4829\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 61 val_rmse: 0.4659\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 62 val_rmse: 0.4621\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 63 val_rmse: 0.4598\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 64 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4575 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 65 val_rmse: 0.4561\n",
            "New best_val_rmse: 0.4561\n",
            "\n",
            "1 steps took 0.401 seconds\n",
            "Epoch: 2 batch_num: 66 val_rmse: 0.4552\n",
            "New best_val_rmse: 0.4552\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 67 val_rmse: 0.4548\n",
            "New best_val_rmse: 0.4548\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 68 val_rmse: 0.4547\n",
            "New best_val_rmse: 0.4547\n",
            "\n",
            "1 steps took 0.395 seconds\n",
            "Epoch: 2 batch_num: 69 val_rmse: 0.4547\n",
            "New best_val_rmse: 0.4547\n",
            "\n",
            "1 steps took 0.395 seconds\n",
            "Epoch: 2 batch_num: 70 val_rmse: 0.4546\n",
            "New best_val_rmse: 0.4546\n",
            "\n",
            "1 steps took 0.396 seconds\n",
            "Epoch: 2 batch_num: 71 val_rmse: 0.4547\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 72 val_rmse: 0.4546\n",
            "New best_val_rmse: 0.4546\n",
            "\n",
            "1 steps took 0.396 seconds\n",
            "Epoch: 2 batch_num: 73 val_rmse: 0.4546\n",
            "New best_val_rmse: 0.4546\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 74 val_rmse: 0.4547\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 75 val_rmse: 0.4549\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 76 val_rmse: 0.4553\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 77 val_rmse: 0.4556\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 78 val_rmse: 0.4557\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 79 val_rmse: 0.4561\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 80 val_rmse: 0.4573\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 81 val_rmse: 0.4589\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 82 val_rmse: 0.4609\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 83 val_rmse: 0.4632\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 84 val_rmse: 0.4648\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 85 val_rmse: 0.4661\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.403 seconds\n",
            "Epoch: 2 batch_num: 86 val_rmse: 0.4668\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 87 val_rmse: 0.4677\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 88 val_rmse: 0.4675\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 89 val_rmse: 0.467\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 90 val_rmse: 0.4667\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 91 val_rmse: 0.4665\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 92 val_rmse: 0.4661\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 93 val_rmse: 0.4662\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 94 val_rmse: 0.4662\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 95 val_rmse: 0.4659\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 96 val_rmse: 0.4658\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.395 seconds\n",
            "Epoch: 2 batch_num: 97 val_rmse: 0.465\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 98 val_rmse: 0.4644\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 99 val_rmse: 0.4638\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 100 val_rmse: 0.4631\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 101 val_rmse: 0.4623\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 102 val_rmse: 0.4614\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 103 val_rmse: 0.4605\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 104 val_rmse: 0.4598\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.398 seconds\n",
            "Epoch: 2 batch_num: 105 val_rmse: 0.4594\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 106 val_rmse: 0.4588\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 107 val_rmse: 0.4584\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 108 val_rmse: 0.4581\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 109 val_rmse: 0.4579\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.397 seconds\n",
            "Epoch: 2 batch_num: 110 val_rmse: 0.4577\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 111 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 112 val_rmse: 0.4575\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 113 val_rmse: 0.4574\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 114 val_rmse: 0.4574\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 115 val_rmse: 0.4574\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 116 val_rmse: 0.4574\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 117 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 118 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 119 val_rmse: 0.4575\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 120 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 121 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 122 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 123 val_rmse: 0.4576\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 124 val_rmse: 0.4577\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 125 val_rmse: 0.4577\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 126 val_rmse: 0.4577\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 127 val_rmse: 0.4578\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 128 val_rmse: 0.4578\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 129 val_rmse: 0.4579\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 130 val_rmse: 0.4579\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 131 val_rmse: 0.4579\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.394 seconds\n",
            "Epoch: 2 batch_num: 132 val_rmse: 0.4579\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.393 seconds\n",
            "Epoch: 2 batch_num: 133 val_rmse: 0.4579\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 134 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 135 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 136 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 137 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 138 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.392 seconds\n",
            "Epoch: 2 batch_num: 139 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "1 steps took 0.391 seconds\n",
            "Epoch: 2 batch_num: 140 val_rmse: 0.458\n",
            "Still best_val_rmse: 0.4546 (from epoch 2)\n",
            "\n",
            "Performance estimates:\n",
            "[0.471089345660429, 0.4791831833542291, 0.45462613367895494]\n",
            "Mean: 0.46829955423120434\n",
            "\n",
            "Fold 4/5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "16 steps took 6.83 seconds\n",
            "Epoch: 0 batch_num: 16 val_rmse: 1.059\n",
            "New best_val_rmse: 1.059\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 32 val_rmse: 0.9468\n",
            "New best_val_rmse: 0.9468\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 48 val_rmse: 0.6678\n",
            "New best_val_rmse: 0.6678\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 64 val_rmse: 0.7191\n",
            "Still best_val_rmse: 0.6678 (from epoch 0)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 0 batch_num: 80 val_rmse: 0.5933\n",
            "New best_val_rmse: 0.5933\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 96 val_rmse: 0.5677\n",
            "New best_val_rmse: 0.5677\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 112 val_rmse: 0.5987\n",
            "Still best_val_rmse: 0.5677 (from epoch 0)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 128 val_rmse: 0.5183\n",
            "New best_val_rmse: 0.5183\n",
            "\n",
            "16 steps took 6.49 seconds\n",
            "Epoch: 1 batch_num: 3 val_rmse: 0.5294\n",
            "Still best_val_rmse: 0.5183 (from epoch 0)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 19 val_rmse: 0.4895\n",
            "New best_val_rmse: 0.4895\n",
            "\n",
            "4 steps took 1.59 seconds\n",
            "Epoch: 1 batch_num: 23 val_rmse: 0.5043\n",
            "Still best_val_rmse: 0.4895 (from epoch 1)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 1 batch_num: 39 val_rmse: 0.4801\n",
            "New best_val_rmse: 0.4801\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 1 batch_num: 43 val_rmse: 0.5345\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 1 batch_num: 59 val_rmse: 0.506\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 75 val_rmse: 0.4955\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 1 batch_num: 83 val_rmse: 0.4979\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 1 batch_num: 91 val_rmse: 0.5505\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 107 val_rmse: 0.5174\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "16 steps took 6.32 seconds\n",
            "Epoch: 1 batch_num: 123 val_rmse: 0.4915\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 1 batch_num: 131 val_rmse: 0.4824\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 1 batch_num: 135 val_rmse: 0.5234\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "16 steps took 6.48 seconds\n",
            "Epoch: 2 batch_num: 10 val_rmse: 0.5056\n",
            "Still best_val_rmse: 0.4801 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 26 val_rmse: 0.473\n",
            "New best_val_rmse: 0.473\n",
            "\n",
            "2 steps took 0.793 seconds\n",
            "Epoch: 2 batch_num: 28 val_rmse: 0.4721\n",
            "New best_val_rmse: 0.4721\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 2 batch_num: 30 val_rmse: 0.4703\n",
            "New best_val_rmse: 0.4703\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 2 batch_num: 32 val_rmse: 0.4731\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 34 val_rmse: 0.4848\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 38 val_rmse: 0.4944\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 2 batch_num: 46 val_rmse: 0.4734\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 48 val_rmse: 0.4762\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.793 seconds\n",
            "Epoch: 2 batch_num: 50 val_rmse: 0.4821\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 54 val_rmse: 0.4821\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 58 val_rmse: 0.477\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 60 val_rmse: 0.4736\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 2 batch_num: 62 val_rmse: 0.4711\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 64 val_rmse: 0.4709\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 66 val_rmse: 0.4715\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 68 val_rmse: 0.4732\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 70 val_rmse: 0.4771\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.792 seconds\n",
            "Epoch: 2 batch_num: 72 val_rmse: 0.4823\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 76 val_rmse: 0.4927\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "8 steps took 3.16 seconds\n",
            "Epoch: 2 batch_num: 84 val_rmse: 0.482\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 88 val_rmse: 0.4769\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 90 val_rmse: 0.4755\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 92 val_rmse: 0.4749\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 94 val_rmse: 0.4744\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 96 val_rmse: 0.4753\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 98 val_rmse: 0.4763\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 100 val_rmse: 0.4773\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.79 seconds\n",
            "Epoch: 2 batch_num: 102 val_rmse: 0.4786\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 104 val_rmse: 0.4792\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.789 seconds\n",
            "Epoch: 2 batch_num: 106 val_rmse: 0.4805\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 110 val_rmse: 0.4826\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 114 val_rmse: 0.4832\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 118 val_rmse: 0.4825\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 122 val_rmse: 0.4815\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 126 val_rmse: 0.4805\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "4 steps took 1.58 seconds\n",
            "Epoch: 2 batch_num: 130 val_rmse: 0.4799\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 132 val_rmse: 0.4797\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.788 seconds\n",
            "Epoch: 2 batch_num: 134 val_rmse: 0.4795\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.791 seconds\n",
            "Epoch: 2 batch_num: 136 val_rmse: 0.4795\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.786 seconds\n",
            "Epoch: 2 batch_num: 138 val_rmse: 0.4794\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "2 steps took 0.787 seconds\n",
            "Epoch: 2 batch_num: 140 val_rmse: 0.4794\n",
            "Still best_val_rmse: 0.4703 (from epoch 2)\n",
            "\n",
            "Performance estimates:\n",
            "[0.471089345660429, 0.4791831833542291, 0.45462613367895494, 0.4702727663542476]\n",
            "Mean: 0.4687928572619652\n",
            "\n",
            "Fold 5/5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "16 steps took 6.84 seconds\n",
            "Epoch: 0 batch_num: 16 val_rmse: 0.8846\n",
            "New best_val_rmse: 0.8846\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 32 val_rmse: 0.7855\n",
            "New best_val_rmse: 0.7855\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 48 val_rmse: 0.6569\n",
            "New best_val_rmse: 0.6569\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 64 val_rmse: 0.6464\n",
            "New best_val_rmse: 0.6464\n",
            "\n",
            "16 steps took 6.35 seconds\n",
            "Epoch: 0 batch_num: 80 val_rmse: 0.6297\n",
            "New best_val_rmse: 0.6297\n",
            "\n",
            "16 steps took 6.45 seconds\n",
            "Epoch: 0 batch_num: 96 val_rmse: 0.569\n",
            "New best_val_rmse: 0.569\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 112 val_rmse: 0.5649\n",
            "New best_val_rmse: 0.5649\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 0 batch_num: 128 val_rmse: 0.5454\n",
            "New best_val_rmse: 0.5454\n",
            "\n",
            "16 steps took 6.48 seconds\n",
            "Epoch: 1 batch_num: 3 val_rmse: 0.5239\n",
            "New best_val_rmse: 0.5239\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 19 val_rmse: 0.5559\n",
            "Still best_val_rmse: 0.5239 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 35 val_rmse: 0.5328\n",
            "Still best_val_rmse: 0.5239 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 51 val_rmse: 0.5344\n",
            "Still best_val_rmse: 0.5239 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 67 val_rmse: 0.5776\n",
            "Still best_val_rmse: 0.5239 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 83 val_rmse: 0.5214\n",
            "New best_val_rmse: 0.5214\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 99 val_rmse: 0.5347\n",
            "Still best_val_rmse: 0.5214 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 115 val_rmse: 0.5271\n",
            "Still best_val_rmse: 0.5214 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 1 batch_num: 131 val_rmse: 0.5265\n",
            "Still best_val_rmse: 0.5214 (from epoch 1)\n",
            "\n",
            "16 steps took 6.48 seconds\n",
            "Epoch: 2 batch_num: 6 val_rmse: 0.53\n",
            "Still best_val_rmse: 0.5214 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 22 val_rmse: 0.527\n",
            "Still best_val_rmse: 0.5214 (from epoch 1)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 38 val_rmse: 0.5194\n",
            "New best_val_rmse: 0.5194\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 54 val_rmse: 0.5311\n",
            "Still best_val_rmse: 0.5194 (from epoch 2)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 70 val_rmse: 0.5279\n",
            "Still best_val_rmse: 0.5194 (from epoch 2)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 86 val_rmse: 0.5142\n",
            "New best_val_rmse: 0.5142\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 102 val_rmse: 0.5134\n",
            "New best_val_rmse: 0.5134\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 118 val_rmse: 0.5159\n",
            "Still best_val_rmse: 0.5134 (from epoch 2)\n",
            "\n",
            "16 steps took 6.33 seconds\n",
            "Epoch: 2 batch_num: 134 val_rmse: 0.5171\n",
            "Still best_val_rmse: 0.5134 (from epoch 2)\n",
            "\n",
            "Performance estimates:\n",
            "[0.471089345660429, 0.4791831833542291, 0.45462613367895494, 0.4702727663542476, 0.5134322286999178]\n",
            "Mean: 0.47772073154955574\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9PqamPNXV5k"
      },
      "source": [
        "# Inference"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7bAMT9-NXV5k"
      },
      "source": [
        "test_dataset = LitDataset(test_df, inference_only=True)"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z5Y_L0yfXV5k",
        "outputId": "ad321b6c-bd60-4600-bcfa-ff3a4e702b14"
      },
      "source": [
        "all_predictions = np.zeros((len(list_val_rmse), len(test_df)))\n",
        "\n",
        "test_dataset = LitDataset(test_df, inference_only=True)\n",
        "test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE,\n",
        "                         drop_last=False, shuffle=False, num_workers=2)\n",
        "\n",
        "for index in range(len(list_val_rmse)):            \n",
        "    model_path = f\"model_{index + 1}.pth\"\n",
        "    print(f\"\\nUsing {model_path}\")\n",
        "                        \n",
        "    model = LitModel()\n",
        "    model.load_state_dict(torch.load(model_path))    \n",
        "    model.to(DEVICE)\n",
        "    \n",
        "    all_predictions[index] = predict(model, test_loader)\n",
        "    \n",
        "    del model\n",
        "    gc.collect()"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Using model_1.pth\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Using model_2.pth\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Using model_3.pth\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Using model_4.pth\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Using model_5.pth\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaModel were not initialized from the model checkpoint at ../input/commonlitreadabilityprize/pretrained-model/clrp_roberta_base/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oUDbpVq8XV5l",
        "outputId": "d4c05531-7613-4dce-99b8-5d5de44f9275"
      },
      "source": [
        "predictions = all_predictions.mean(axis=0)\n",
        "submission_df.target = predictions\n",
        "print(submission_df)\n",
        "submission_df.to_csv(\"submission.csv\", index=False)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "          id    target\n",
            "0  c0f722661 -0.404446\n",
            "1  f0953f0a5 -0.597980\n",
            "2  0df072751 -0.258812\n",
            "3  04caf4e0c -2.492555\n",
            "4  0e63f8bea -1.670199\n",
            "5  12537fe78 -1.504853\n",
            "6  965e592c0  0.083787\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uXCmIyftjlEY"
      },
      "source": [
        "# Upload data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3scDIDWzJu_X",
        "outputId": "4670c9c5-798e-4956-da48-0d3dbbf35d42"
      },
      "source": [
        "!date +\"%Y%m%d%I%M%S\""
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "20210704055711\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X8IUhtyYjmso",
        "outputId": "4e2a017f-6dcc-435d-e596-74eedd799334"
      },
      "source": [
        "!mkdir -p ./output/\n",
        "!cp -f ./model* ./output/\n",
        "#CHANGEME\n",
        "!cp -f ./drive/MyDrive/kaggle/commonlit/Lightweight-Roberta-base/dataset-metadata.json ./output/dataset-metadata.json\n",
        "!sed -i -e \"s/lightweight-roberta-base/lightweight-roberta-base-`date +\"%Y%m%d%I%M%S\"`/\" ./output/dataset-metadata.json\n",
        "!kaggle datasets create -p ./output/"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Starting upload for file model_3.pth\n",
            "Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.12 / client 1.5.4)\n",
            "100% 477M/477M [00:10<00:00, 46.1MB/s]\n",
            "Upload successful: model_3.pth (477MB)\n",
            "Starting upload for file model_5.pth\n",
            "100% 477M/477M [00:12<00:00, 40.8MB/s]\n",
            "Upload successful: model_5.pth (477MB)\n",
            "Starting upload for file model_4.pth\n",
            "100% 477M/477M [00:09<00:00, 53.3MB/s]\n",
            "Upload successful: model_4.pth (477MB)\n",
            "Starting upload for file model_2.pth\n",
            "100% 477M/477M [00:11<00:00, 44.2MB/s]\n",
            "Upload successful: model_2.pth (477MB)\n",
            "Starting upload for file model_1.pth\n",
            "100% 477M/477M [00:11<00:00, 42.5MB/s]\n",
            "Upload successful: model_1.pth (477MB)\n",
            "Your private Dataset is being created. Please check progress at /api/v1/datasets/status//iamnishipy/lightweight-roberta-base-20210704055800\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uWMDce0OKDuQ",
        "outputId": "8f6d65da-88c6-4663-a744-c5ea46103aa7"
      },
      "source": [
        "!cat ./output/dataset-metadata.json"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"licenses\": [\n",
            "    {\n",
            "      \"name\": \"CC0-1.0\"\n",
            "    }\n",
            "  ], \n",
            "  \"id\": \"iamnishipy/lightweight-roberta-base-20210704055731\", \n",
            "  \"title\": \"Lightweight-Roberta-base\"\n",
            "}"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}