prompting_example.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "### Define the Prompt\n",
        "\n",
        "We define a prompt that can perform two tasks given a topic and an argument <br>\n",
        "1) classify whether the argument is a fallacy, and if so what fallacy the  argument contains <br>\n",
        "2) decide whether the argument supports or refutes the topic <br>\n",
        "Prompts templates are defined by a string where ```{}``` is a placeholder for where the inputs for a sample are to be and ```<mask>``` is filled in as the predicted label. <br>\n",
        "*Multiple masked tokens in a single prompt is currently not supported*"
      ],
      "metadata": {
        "id": "y5Mq5nlxdlNu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "templates = {\"fallacy\": \"fallacy task. Topic: {} Text: {} This contains the fallacy: <mask>\", \n",
        "             \"stance\": \"procon task. Topic: {} Text: {} Has the relation: <mask>\"}\n",
        "\n",
        "def fallacy_policy(pred):\n",
        "  fallacies = {'AppealtoEmotion', 'RedHerring', 'NoFallacy', 'IrrelevantAuthority','AdHominem','HastyGeneralization'}\n",
        "  if pred not in fallacies: return 'UNKNOWN'\n",
        "  return pred\n",
        "def stance_policy(pred):\n",
        "  if pred not in {\"support\", \"contradict\"}: return \"UNKNOWN\"\n",
        "  return pred\n",
        "\n",
        "policies = {\"fallacy\": fallacy_policy, \"stance\": stance_policy}\n",
        "\n",
        "argument_prompt = Prompt(templates, policies)"
      ],
      "metadata": {
        "id": "ZLIUPefZB8Bu"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "RobertaPrompt uses this class to convert a sample into it's desire prompt - for example"
      ],
      "metadata": {
        "id": "7GnB9BUHfT0o"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "argument_prompt.test_sample([\"Should we allow animal testing?\", \"Animal testing abuses animals and should be dis-continued\"], \"fallacy\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "id": "GXLu5H6hfR85",
        "outputId": "9fb31b95-1916-4e09-a002-aef89daf8efd"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'fallacy task. Topic: Should we allow animal testing? Text: Animal testing abuses animals and should be dis-continued This contains the fallacy: <mask>'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "argument_prompt.train_sample([\"Should we allow animal testing?\", \"Animal testing abuses animals and should be dis-continued\"], \"NoFallacy\", \"fallacy\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "id": "VsjncXpYbyEN",
        "outputId": "00b6ee07-f32f-4544-86ec-96c1e7129317"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'fallacy task. Topic: Should we allow animal testing? Text: Animal testing abuses animals and should be dis-continued This contains the fallacy: NoFallacy'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(argument_prompt)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lpl-ALGP6BVB",
        "outputId": "c6891433-8bc3-4d58-f689-1cdc312ede84"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "=== Prompts ===\n",
            "Task: fallacy, Template: fallacy task. Topic: {} Text: {} This contains the fallacy: <mask>\n",
            "Task: stance, Template: procon task. Topic: {} Text: {} Has the relation: <mask>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Load Model\n",
        "We load a model we have already trained for this task. Some sample predictions are also displayed"
      ],
      "metadata": {
        "id": "mEgFVKDueKMd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pmodel = RobertaPrompt(model='/content/drive/MyDrive/Laidlaw Research Project/models/prompt_combined', device = torch.device('cuda'), prompt = argument_prompt)"
      ],
      "metadata": {
        "id": "eZn5mZMuQQjz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(pmodel)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AwiHWWR_6rzS",
        "outputId": "28aca836-87c1-48e8-b411-9ff231a8857a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "======== Base Model ============\n",
            "/content/drive/MyDrive/Laidlaw Research Project/models/prompt_combined\n",
            "\n",
            "======== Tasks ============\n",
            "Task: fallacy, Template: fallacy task. Topic: {} Text: {} This contains the fallacy: <mask>\n",
            "Task: stance, Template: procon task. Topic: {} Text: {} Has the relation: <mask>\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fallacy = pmodel.infer([\"Should we allow animal testing?\", \"Animal testing abuses animals and should be dis-continued\"], \"fallacy\")\n",
        "stance = pmodel.infer([\"Should we allow animal testing?\", \"Animal testing abuses animals and should be dis-continued\"], \"stance\")\n",
        "print(\"Fallacy: {}\\nStance: {}\".format(fallacy, stance))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CLkf_WLyW8zg",
        "outputId": "98dade76-810f-40b9-96e6-ac401a8c6d78"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fallacy: NoFallacy\n",
            "Stance: contradict\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fallacy = pmodel.infer([\"Should we allow animal testing?\", \"Your stupid for bringing this up, animal testing is horrible\"], \"fallacy\")\n",
        "stance = pmodel.infer([\"Should we allow animal testing?\", \"Your stupid for bringing this up, animal testing is horrible\"], \"stance\")\n",
        "print(\"Fallacy: {}\\nStance: {}\".format(fallacy, stance))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5MRzsviaenxX",
        "outputId": "34621a83-8c55-4444-9959-41efd146fdac"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fallacy: AdHominem\n",
            "Stance: contradict\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fallacy = pmodel.infer([\"Should we allow animal testing?\", \"My Dad had a dog once, he says animal testing should be allowed\"], \"fallacy\")\n",
        "stance = pmodel.infer([\"Should we allow animal testing?\", \"My Dad had a dog once, he says animal testing should be allowed\"], \"stance\")\n",
        "print(\"Fallacy: {}\\nStance: {}\".format(fallacy, stance))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "58wuvAv3evt9",
        "outputId": "0c397653-4442-4791-dfce-4fd4b273d9b7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fallacy: IrrelevantAuthority\n",
            "Stance: support\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fallacy = pmodel.infer([\"Should we allow animal testing?\", \"Everyone has a favorite animal, what is yours?\"], \"fallacy\")\n",
        "stance = pmodel.infer([\"Should we allow animal testing?\", \"Everyone has a favorite animal, what is yours?\"], \"stance\")\n",
        "print(\"Fallacy: {}\\nStance: {}\".format(fallacy, stance))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l77z7efUe4xX",
        "outputId": "053712bd-5b39-438d-ba9f-df0d52fd30b5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fallacy: RedHerring\n",
            "Stance: support\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(pmodel.test(\"/content/drive/MyDrive/Laidlaw Research Project/data/test_samples.tsv\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jfPTCfwbXdSe",
        "outputId": "e5b2c4d5-36a6-4728-a215-c20ef86f1399"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "macro f1: 0.7863247863247864\n",
            "micro f1: 0.7933884297520661\n",
            "weighted f1: 0.7943516538557861\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "  contradict       0.73      0.77      0.75        48\n",
            "     support       0.84      0.81      0.83        73\n",
            "\n",
            "    accuracy                           0.79       121\n",
            "   macro avg       0.78      0.79      0.79       121\n",
            "weighted avg       0.80      0.79      0.79       121\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Train Model"
      ],
      "metadata": {
        "id": "NUwprAyYiJmm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pmodel = RobertaPrompt(model='roberta-large', device = torch.device('cuda'), prompt = argument_prompt)\n",
        "print(pmodel)"
      ],
      "metadata": {
        "id": "Fs2FJwbmiKwM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "99366463-4335-4efc-a0b7-9ee64bbe03f9"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "======== Base Model ============\n",
            "roberta-large\n",
            "\n",
            "======== Tasks ============\n",
            "Task: fallacy, Template: fallacy task. Topic: {} Text: {} This contains the fallacy: <mask>\n",
            "Task: stance, Template: procon task. Topic: {} Text: {} Has the relation: <mask>\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pmodel.train(\"/content/drive/MyDrive/Laidlaw Research Project/data/stance/training.tsv\", \"/content/drive/MyDrive/Laidlaw Research Project/data/stance/val.tsv\", output_dir=\"/content/drive/MyDrive/Laidlaw Research Project/data/stance/model\", epochs=3)"
      ],
      "metadata": {
        "id": "BkYSvBpmnwCy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08c4ec67-67da-4466-b0b3-e5f0c0e37cb8"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "======== Epoch 1 / 3 ========\n",
            "Training...\n",
            "  Batch    10  of     62.    Elapsed: 0:00:11.\n",
            "  Batch    20  of     62.    Elapsed: 0:00:19.\n",
            "  Batch    30  of     62.    Elapsed: 0:00:27.\n",
            "  Batch    40  of     62.    Elapsed: 0:00:35.\n",
            "  Batch    50  of     62.    Elapsed: 0:00:43.\n",
            "  Batch    60  of     62.    Elapsed: 0:00:52.\n",
            "\n",
            "  Average training loss: 2.67\n",
            "  Training epcoh took: 0:00:53\n",
            "\n",
            "Running Validation...\n",
            "SAVING NEW MODEL ... \n",
            "  Validation Loss: 0.01\n",
            "  Validation took: 0:00:31\n",
            "\n",
            "======== Epoch 2 / 3 ========\n",
            "Training...\n",
            "  Batch    10  of     62.    Elapsed: 0:00:08.\n",
            "  Batch    20  of     62.    Elapsed: 0:00:16.\n",
            "  Batch    30  of     62.    Elapsed: 0:00:25.\n",
            "  Batch    40  of     62.    Elapsed: 0:00:33.\n",
            "  Batch    50  of     62.    Elapsed: 0:00:41.\n",
            "  Batch    60  of     62.    Elapsed: 0:00:49.\n",
            "\n",
            "  Average training loss: 0.01\n",
            "  Training epcoh took: 0:00:50\n",
            "\n",
            "Running Validation...\n",
            "SAVING NEW MODEL ... \n",
            "  Validation Loss: 0.00\n",
            "  Validation took: 0:00:08\n",
            "\n",
            "======== Epoch 3 / 3 ========\n",
            "Training...\n",
            "  Batch    10  of     62.    Elapsed: 0:00:08.\n",
            "  Batch    20  of     62.    Elapsed: 0:00:17.\n",
            "  Batch    30  of     62.    Elapsed: 0:00:25.\n",
            "  Batch    40  of     62.    Elapsed: 0:00:33.\n",
            "  Batch    50  of     62.    Elapsed: 0:00:41.\n",
            "  Batch    60  of     62.    Elapsed: 0:00:49.\n",
            "\n",
            "  Average training loss: 0.00\n",
            "  Training epcoh took: 0:00:50\n",
            "\n",
            "Running Validation...\n",
            "SAVING NEW MODEL ... \n",
            "  Validation Loss: 0.00\n",
            "  Validation took: 0:00:09\n",
            "\n",
            "Training complete!\n",
            "Total training took 0:03:22 (h:mm:ss)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{1: {'Training Loss': 2.665111167721578,\n",
              "  'Valid. Loss': 0.0055667326669208705,\n",
              "  'Training Time': '0:00:53',\n",
              "  'Validation Time': '0:00:31'},\n",
              " 2: {'Training Loss': 0.00734625298965482,\n",
              "  'Valid. Loss': 0.00396701754652895,\n",
              "  'Training Time': '0:00:50',\n",
              "  'Validation Time': '0:00:08'},\n",
              " 3: {'Training Loss': 0.004069779668146024,\n",
              "  'Valid. Loss': 0.003757576982025057,\n",
              "  'Training Time': '0:00:50',\n",
              "  'Validation Time': '0:00:09'}}"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(pmodel.test(\"/content/drive/MyDrive/Laidlaw Research Project/data/stance/test_samples.tsv\"))"
      ],
      "metadata": {
        "id": "7o3GJocynwZg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "954351b8-fb15-4714-f5f9-654d503dd325"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "macro f1: 0.7993366500829187\n",
            "micro f1: 0.8016528925619834\n",
            "weighted f1: 0.8037909625426586\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "  contradict       0.70      0.88      0.78        48\n",
            "     support       0.90      0.75      0.82        73\n",
            "\n",
            "    accuracy                           0.80       121\n",
            "   macro avg       0.80      0.81      0.80       121\n",
            "weighted avg       0.82      0.80      0.80       121\n",
            "\n"
          ]
        }
      ]
    }
  ]
}