|
-||[Simple Geometric (circles and triangles)](https://doi.org/10.5281/zenodo.5012824)| Binary shape *classificaiton* |
|
-||[Imagenet](https://image-net.org/download.php) |$1000$ classes natural images *classificaiton* | 
|
-|*Text*| [Stanford sentiment treebank](https://nlp.stanford.edu/sentiment/index.html) |Positive or negative movie reviews sentiment *classificaiton* | 
|
-|*Timeseries* | [Coffee dataset](https://www.timeseriesclassification.com/description.php?Dataset=Coffee) | Binary *classificaiton* of Robusta and Aribica coffee beans| |
+||[Simple Geometric (circles and triangles)](https://doi.org/10.5281/zenodo.5012824)| Binary shape *classificaiton* ||
+||[Imagenet](https://image-net.org/download.php) | $1000$ classes natural images *classificaiton* | |
+|*Text*| [Stanford sentiment treebank](https://nlp.stanford.edu/sentiment/index.html) | Positive or negative movie reviews sentiment *classification* | |
+|*Timeseries* | [Coffee dataset](https://www.timeseriesclassification.com/description.php?Dataset=Coffee) | Binary *classificaiton* of Robusta and Aribica coffee beans | 
|
-|*Text*| | | |
+|*Text*| [EU-law statements](https://zenodo.org/records/8200001) | Regulatory or non-regulatory *classification* | 
|
|*Timeseries* | Fast Radio Burst (FRB) dataset (not publicly available) | Binary *classificaiton* of Fast Radio Burst (FRB) timeseries data : noise or a real FRB. | ](./explainers/LIME/lime_images.ipynb) or [](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_images.ipynb) ||
-|*Text* | | | |
-| *Time series*| [](./explainers/LIME/lime_images.ipynb) or [](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_images.ipynb) ||
+| | | [](./explainers/LIME/lime_text_eulaw.ipynb) or [](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_text_eulaw.ipynb) | |
+| *Time series* | [ (*i*) | (*i*) |](./explainers/LIME/lime_text_eulaw.ipynb) |
+| ------------- | ------------- |--------| -----| -----|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| $n_{samples}$ | **$5000$** | $1000$ | $10 000$| $500$| 2000 |
+| *Kernel Width* | **$25$**| default | default| default| default |
+| $n_{features}$ | **$10$** | $30$ | default| default| 999 |
-🠊 The most crucial parameter is the *Kernel width*: low values cause high sensitivity, however that observaiton was dependant on the evaluaiton metric.
+🠊 The most crucial parameter is the *Kernel width*: low values cause high sensitivity, however that observation was dependent on the evaluation metric.
#### KernelSHAP
| Hyperparameter | Default value | (*i*)| (*i*) | ![\"Logo_ER10\"](\"https://user-images.githubusercontent.com/3244249/151994514-b584b984-a148-4ade-80ee-0f88b0aefa45.png\")
\n",
+ "\n",
+ "### Interpreting the sentence classification of legal texts using LIME\n",
+ "\n",
+ "LIME (Local Interpretable Model-agnostic Explanations) is an explainable-AI method that aims to create an interpretable model that locally represents the classifier. For more details see the [LIME paper](https://arxiv.org/abs/1602.04938).\n",
+ "\n",
+ "This notebook demonstrates how to use the LIME explainable-AI method in [DIANNA](https://github.com/dianna-ai/dianna) to explain a text classification model created as part of the [Nature of EU Rules project](https://research-software-directory.org/projects/the-nature-of-eu-rules-strict-and-detailed-or-lacking-bite). The model is used to perform binary classification of individual sentences from EU legislation to determine whether they specify a regulation or not (i.e., whether they specify a legal obligation or prohibition that some legal entity should comply with). [Here's an example](https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32012R1215&qid=1724343987254) of what an EU legislative document looks like.\n",
+ "\n",
+ "##### Regulatory sentence example:\n",
+ "\n",
+ "```Citizens of all Member States shall separate their recyclables before disposing of trash, or else face a fine.```\n",
+ "\n",
+ "##### Non-regulatory (constitutive) sentence example:\n",
+ "\n",
+ "```This Regulation shall apply in civil and commercial matters whatever the nature of the court or tribunal.```\n",
+ "\n",
+ "**Note:** while the occurrence of words like ``shall`` and ``must`` (which are called **deontic** words) are necessary condition for a sentence to be classified as regulatory, they are not a sufficient condition."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fa6d17b0",
+ "metadata": {},
+ "source": [
+ "#### Colab Setup"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "471630ff",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:13:59.624761Z",
+ "start_time": "2024-07-10T11:13:59.563650Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "running_in_colab = 'google.colab' in str(get_ipython())\n",
+ "if running_in_colab:\n",
+ " # install dianna\n",
+ " !python3 -m pip install dianna[notebooks]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a5cf6f82-c1c7-4814-ae0f-5a1c0b8578f6",
+ "metadata": {},
+ "source": [
+ "#### 0. Imports and paths"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "34b556d8-5337-44dc-8efe-14d1dff6f011",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:13:59.625762Z",
+ "start_time": "2024-07-10T11:13:59.568658Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "C:\\Users\\ChristiaanMeijer\\anaconda3\\envs\\dianna3112\\Lib\\site-packages\\torchtext\\data\\__init__.py:4: UserWarning: \n",
+ "/!\\ IMPORTANT WARNING ABOUT TORCHTEXT STATUS /!\\ \n",
+ "Torchtext is deprecated and the last released version will be 0.18 (this one). You can silence this warning by calling the following at the beginnign of your scripts: `import torchtext; torchtext.disable_torchtext_deprecation_warning()`\n",
+ " warnings.warn(torchtext._TORCHTEXT_DEPRECATION_MSG)\n"
+ ]
+ }
+ ],
+ "source": [
+ "from torch.utils.data import DataLoader\n",
+ "from typing import Iterable\n",
+ "from tqdm import tqdm\n",
+ "\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import torch\n",
+ "import xgboost\n",
+ "\n",
+ "import dianna\n",
+ "from dianna import visualization\n",
+ "from dianna.utils.downloader import download\n",
+ "from dianna.utils.tokenizers import SpacyTokenizer"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "c616916c-78ef-48d0-a744-b25b37b62a3f",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:13:59.666220Z",
+ "start_time": "2024-07-10T11:13:59.576245Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "class_names = ['constitutive', 'regulatory']\n",
+ "model_path = download('inlegal_bert_xgboost_classifier.json', 'model')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "# 1 - Define test data"
+ ],
+ "id": "156805d0e3de333b"
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:13:59.666220Z",
+ "start_time": "2024-07-10T11:13:59.590692Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "constitutive_statement_0 = \"The purchase, import or transport from Syria of crude oil and petroleum products shall be prohibited.\"\n",
+ "constitutive_statement_1 = \"This Decision shall enter into force on the twentieth day following that of its publication in the Official Journal of the European Union.\"\n",
+ "regulatory_statement_0 = \"Where observations are submitted, or where substantial new evidence is presented, the Council shall review its decision and inform the person or entity concerned accordingly.\"\n",
+ "regulatory_statement_1 = \"The relevant Member State shall inform the other Member States of any authorisation granted under this Article.\"\n",
+ "regulatory_statement_2 = \"Member States shall cooperate, in accordance with their national legislation, with inspections and disposals undertaken pursuant to paragraphs 1 and 2.\""
+ ],
+ "id": "200b1ed56f4ddf5f"
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bad4f5b1-6097-4ef3-98c4-78432ad640b0",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-03-21T10:28:34.466985Z",
+ "start_time": "2024-03-21T10:28:34.456937Z"
+ }
+ },
+ "source": [
+ "# 2 - Load and prepare the model\n",
+ "\n",
+ "The model is a combination of a pretrained transformer used as a feature extractor, with an XGBoost model trained on top. The following cells load the model into the variable called `model_runner`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:13:59.666763Z",
+ "start_time": "2024-07-10T11:13:59.598468Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "from transformers import AutoTokenizer, AutoModel\n",
+ "def create_features(texts: list[str], model_tag=\"law-ai/InLegalBERT\") -> torch.Tensor:\n",
+ " \"\"\"Create features for a list of texts.\"\"\"\n",
+ " max_length = 512\n",
+ " tokenizer = AutoTokenizer.from_pretrained(model_tag)\n",
+ " model = AutoModel.from_pretrained(model_tag)\n",
+ "\n",
+ " def process_batch(batch: Iterable[str]):\n",
+ " cropped_texts = [text[:max_length] for text in batch]\n",
+ " encoded_inputs = tokenizer(cropped_texts, padding='longest', truncation=True, max_length=max_length,\n",
+ " return_tensors=\"pt\")\n",
+ " with torch.no_grad():\n",
+ " outputs = model(**encoded_inputs)\n",
+ " last_hidden_states = outputs.last_hidden_state\n",
+ " sentence_features = last_hidden_states.mean(dim=1)\n",
+ " return sentence_features\n",
+ "\n",
+ " dataloader = DataLoader(texts, batch_size=1) # batch size of 1 was quickest for my development machine\n",
+ " features = [process_batch(batch) for batch in tqdm(dataloader, desc=f'Creating features')]\n",
+ " return np.array(torch.cat(features, dim=0))\n",
+ "\n"
+ ],
+ "id": "e1a16e955d860d89"
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:13:59.672763Z",
+ "start_time": "2024-07-10T11:13:59.606453Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "models={}\n",
+ "def classify_texts(texts: list[str], model_path, return_proba: bool = False):\n",
+ " \"\"\"Classifies every text in a list of texts using the xgboost model stored in model_path.\n",
+ "\n",
+ " The xgboost model will be loaded and used to classify the texts. The texts however will first be processed by a\n",
+ " large language model which will do the feature extraction for every text. The classifications of the\n",
+ " xgboost model will be returned.\n",
+ " For training the xgboost model, see train_legalbert_xgboost.py.\n",
+ "\n",
+ " Parameters\n",
+ " ----------\n",
+ " texts\n",
+ " A list of strings of which each needs to be classified.\n",
+ " model_path\n",
+ " The path to a stored xgboost model\n",
+ " return_proba\n",
+ " return the probabilities of the model\n",
+ "\n",
+ " Returns\n",
+ " -------\n",
+ " List of classifications, one for every text in the list\n",
+ "\n",
+ " \"\"\"\n",
+ " features = create_features(texts)\n",
+ " if model_path not in models:\n",
+ " print(f'Loading model from {model_path}.')\n",
+ " model = xgboost.XGBClassifier()\n",
+ " model.load_model(model_path)\n",
+ " models[model_path] = model\n",
+ "\n",
+ " model = models[model_path]\n",
+ " if return_proba:\n",
+ " return model.predict_proba(features)\n",
+ " return model.predict(features)"
+ ],
+ "id": "9d8346a5f0e78d5d"
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "555842c5-3f82-4f63-93bb-696645d4b447",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:14:00.114564Z",
+ "start_time": "2024-07-10T11:13:59.617764Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "class StatementClassifier:\n",
+ " def __init__(self):\n",
+ " self.tokenizer = SpacyTokenizer(name='en_core_web_sm')\n",
+ "\n",
+ " def __call__(self, sentences):\n",
+ " # ensure the input has a batch axis\n",
+ " if isinstance(sentences, str):\n",
+ " sentences = [sentences]\n",
+ "\n",
+ " probs = classify_texts(sentences, model_path, return_proba=True)\n",
+ "\n",
+ " return np.transpose([(probs[:, 0]), (1 - probs[:, 0])])\n",
+ "\n",
+ "model_runner = StatementClassifier()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## Test the model"
+ ],
+ "id": "e6ae5f1011540179"
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2024-07-10T11:14:01.218604Z",
+ "start_time": "2024-07-10T11:14:00.117563Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Creating features: 100%|██████████| 5/5 [00:01<00:00, 3.63it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Loading model from C:\\Users\\ChristiaanMeijer\\AppData\\Local\\dianna\\dianna\\Cache\\inlegal_bert_xgboost_classifier.json.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | statement | \n", + "prediction | \n", + "actual | \n", + "
|---|---|---|---|
| 0 | \n", + "The purchase, import or transport from Syria o... | \n", + "constitutive | \n", + "constitutive | \n", + "
| 1 | \n", + "This Decision shall enter into force on the tw... | \n", + "constitutive | \n", + "constitutive | \n", + "
| 2 | \n", + "Where observations are submitted, or where sub... | \n", + "regulatory | \n", + "regulatory | \n", + "
| 3 | \n", + "The relevant Member State shall inform the oth... | \n", + "regulatory | \n", + "regulatory | \n", + "
| 4 | \n", + "Member States shall cooperate, in accordance w... | \n", + "regulatory | \n", + "regulatory | \n", + "
| \n", + " | The | \n", + "relevant | \n", + "Member | \n", + "State | \n", + "shall | \n", + "inform | \n", + "the | \n", + "other | \n", + "Member | \n", + "States | \n", + "of | \n", + "any | \n", + "authorisation | \n", + "granted | \n", + "under | \n", + "this | \n", + "Article | \n", + ". | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "3.000000 | \n", + "
| mean | \n", + "0.066543 | \n", + "0.040031 | \n", + "0.060046 | \n", + "0.105916 | \n", + "0.192736 | \n", + "0.145457 | \n", + "0.078587 | \n", + "0.064191 | \n", + "0.089121 | \n", + "0.142354 | \n", + "0.114781 | \n", + "0.042535 | \n", + "0.065503 | \n", + "0.039873 | \n", + "0.091866 | \n", + "0.080870 | \n", + "0.054341 | \n", + "0.111533 | \n", + "
| std | \n", + "0.027477 | \n", + "0.007484 | \n", + "0.022590 | \n", + "0.000665 | \n", + "0.008612 | \n", + "0.014756 | \n", + "0.004525 | \n", + "0.004821 | \n", + "0.011775 | \n", + "0.022780 | \n", + "0.011397 | \n", + "0.013324 | \n", + "0.010638 | \n", + "0.009031 | \n", + "0.003086 | \n", + "0.010070 | \n", + "0.020705 | \n", + "0.005437 | \n", + "
| min | \n", + "0.049605 | \n", + "0.034720 | \n", + "0.038425 | \n", + "0.105155 | \n", + "0.182833 | \n", + "0.128693 | \n", + "0.073430 | \n", + "0.060816 | \n", + "0.075847 | \n", + "0.124168 | \n", + "0.102572 | \n", + "0.028173 | \n", + "0.059008 | \n", + "0.029511 | \n", + "0.088517 | \n", + "0.069802 | \n", + "0.038332 | \n", + "0.108313 | \n", + "
| 25% | \n", + "0.050691 | \n", + "0.035751 | \n", + "0.048323 | \n", + "0.105681 | \n", + "0.189868 | \n", + "0.139946 | \n", + "0.076934 | \n", + "0.061430 | \n", + "0.084527 | \n", + "0.129578 | \n", + "0.109601 | \n", + "0.036556 | \n", + "0.059365 | \n", + "0.036776 | \n", + "0.090501 | \n", + "0.076558 | \n", + "0.042649 | \n", + "0.108395 | \n", + "
| 50% | \n", + "0.051778 | \n", + "0.036782 | \n", + "0.058221 | \n", + "0.106208 | \n", + "0.196902 | \n", + "0.151198 | \n", + "0.080438 | \n", + "0.062044 | \n", + "0.093207 | \n", + "0.134989 | \n", + "0.116630 | \n", + "0.044939 | \n", + "0.059722 | \n", + "0.044042 | \n", + "0.092485 | \n", + "0.083314 | \n", + "0.046967 | \n", + "0.108477 | \n", + "
| 75% | \n", + "0.075011 | \n", + "0.042686 | \n", + "0.070857 | \n", + "0.106296 | \n", + "0.197688 | \n", + "0.153839 | \n", + "0.081166 | \n", + "0.065878 | \n", + "0.095758 | \n", + "0.151447 | \n", + "0.120885 | \n", + "0.049716 | \n", + "0.068751 | \n", + "0.045054 | \n", + "0.093540 | \n", + "0.086404 | \n", + "0.062345 | \n", + "0.113144 | \n", + "
| max | \n", + "0.098245 | \n", + "0.048591 | \n", + "0.083493 | \n", + "0.106385 | \n", + "0.198473 | \n", + "0.156479 | \n", + "0.081894 | \n", + "0.069713 | \n", + "0.098308 | \n", + "0.167905 | \n", + "0.125140 | \n", + "0.054494 | \n", + "0.077780 | \n", + "0.046066 | \n", + "0.094595 | \n", + "0.089493 | \n", + "0.077724 | \n", + "0.117810 | \n", + "
| \n", + " | 0 | \n", + "1 | \n", + "2 | \n", + "
|---|---|---|---|
| 0 | \n", + "The | \n", + "0 | \n", + "0.040727 | \n", + "
| 1 | \n", + "relevant | \n", + "1 | \n", + "0.063082 | \n", + "
| 2 | \n", + "Member | \n", + "2 | \n", + "0.065679 | \n", + "
| 3 | \n", + "State | \n", + "3 | \n", + "0.080698 | \n", + "
| 4 | \n", + "shall | \n", + "4 | \n", + "0.171471 | \n", + "
| 5 | \n", + "inform | \n", + "5 | \n", + "0.142825 | \n", + "
| 6 | \n", + "the | \n", + "6 | \n", + "0.073943 | \n", + "
| 7 | \n", + "other | \n", + "7 | \n", + "0.078633 | \n", + "
| 8 | \n", + "Member | \n", + "8 | \n", + "0.070353 | \n", + "
| 9 | \n", + "States | \n", + "9 | \n", + "0.132782 | \n", + "
| 10 | \n", + "of | \n", + "10 | \n", + "0.115537 | \n", + "
| 11 | \n", + "any | \n", + "11 | \n", + "0.059245 | \n", + "
| 12 | \n", + "authorisation | \n", + "12 | \n", + "0.057643 | \n", + "
| 13 | \n", + "granted | \n", + "13 | \n", + "0.057985 | \n", + "
| 14 | \n", + "under | \n", + "14 | \n", + "0.118352 | \n", + "
| 15 | \n", + "this | \n", + "15 | \n", + "0.085583 | \n", + "
| 16 | \n", + "Article | \n", + "16 | \n", + "0.062079 | \n", + "
| 17 | \n", + ". | \n", + "17 | \n", + "0.120736 | \n", + "